Extension talk:Collection

Previous page history was archived for backup purposes at Extension talk:Collection/LQT Archive 1 on 2015-07-10.

Archives

/Archive 1

Hello. I have tried the mwlib mailing list first but it seems to be rather "dead". We have been using the Collection extension and our own render server for some time now (on Ubuntu Server 12.04 LTS). It generates pdf's and it worke(d) like a charm. We recently encountered a strange problem, we noticed that it was still possible to render a page or book but it was not possible to download the pdf anymore. When you then press "Download the file" the following message pops up:

File not found:

The file you are trying to download does not exist: Maybe it has been deleted and needs to be regenerated.
Return to ........

We have been updating to MW 1.22.0 recently but we noticed this behavior also in MW.1.21.x. To make a long story a little shorter, I can get it working again when I open (reload) our wiki's "main page" in a browser on the Web server itself. After that I can download generated pdf files again from any computer on the network. But after a couple of minutes it stops working again.

In Collection.body.php I found 'coll-download_notfound_text' which is probably responsible for the message:

        $info = false;
        if ( isset( $r['url'] ) ) {
            $req = MWHttpRequest::factory( $r['url'] );
            $req->setCallback( array( $this, 'writeToTempFile' ) );
            if ( $req->execute()->isOK() ) {
                $info = true;
            }
            $content_type = $r['content_type'];
            $content_length = $r['content_length'];
            $content_disposition = $r['content_disposition'];
        } else {
            $info = self::mwServeCommand( 'download', array(
                'collection_id' => $request->getVal( 'collection_id' ),
                'writer' => $request->getVal( 'writer' ),
            ) );
            $content_type = $info['content_type'];
            $content_length = $info['download_content_length'];
            $content_disposition = null;
        }
        if ( !$info ) {
            $this->getOutput()->showErrorPage( 'coll-download_notfound_title', 'coll-download_notfound_text' );
            return;
        }

I am not a programmer but does anyone know what the above code does and in witch situation it will show the message ('coll-download_notfound_text'): 'The file you are trying to download does not exist'?

We are running our Wiki on Windows Server 2008 and are using XCache but we have been using XCache for some time now and never encountered problems like this before. It also makes no difference if I run the Wiki with the caching disabled. I also tried several versions of the Collection extension form Git but to no avail. Any help would be appreciated. Thanks.

I have seen this problem today for the first time at http://espiral.org

We still haven't succeed downloading a proper PDF (not true, I just got one for a stub, next to +20 failed attempts).

I wonder whether such PDF is supposed to be found in the PediaPress servers, or in my host.

This post was posted by Qgil-WMF, but signed as Qgil.

Hi Felipe,

I have the same problem... Did you find a solution?

Thanks

Pivic

Hello Pivic. No I did not find a proper solution yet. I must say that I did not spend or had any time on finding one. Our quick and dirty one was and still is a job that re-load's the main wiki page on the server every 2 minutes. What would be interesting is what the message exactly means. Is the download url "broken"? If so what is wrong with it? Regards.

Hi,

I tried your technique but I don't get it to work... I wonder how to double check the URL.. When I look at it, it does not make sense... Maybe someone can post his working URL to see if they are completely diffrent?

The Download button points to: http://www.WEBSITENAME.co.uk/wiki/index.php?title=Special:Book&bookcmd=download&collection_id=9b69472ee6d59d3e&writer=rl&return_to=ARTICLE_NAME

When I try to download my file from my wiki, the render server shows: 213.251.182.12 - - [2014-09-20 14:35:53] "POST /cache/ HTTP/1.0" 200 496 0.049885

But if I type:

192.168.0.7:8898/cache/9b/9b69472ee6d59d3e/output.rl in the browser on my personnal render server then I can download the PDF....

Then if I replace 192.168.0.7 by what google gives me as "what's my IP", then I can download the file with my phone on 3G.

How can I get the URL from my wiki to point to the actual file correctly?

To carry on the debug, in Collection.body.php I added echo " $url " at line 1164 (in the if(!info){}) and read: http://192.168.0.7:8898/cache/9b/9b69472ee6d59d3e/output.rl I read this from my phone on 3G, from a computer connected to the same router and also if I do it from the render server itself.

This URL works (I can download the rendered file) if I type it in the browser of my server and other computers on the local network. But it wouldn't work from anywhere else (i.e. on 3G)..

I think I should see My Actual IP instead of the 192.168.0.7 local one... And also, I should not be reaching this "if(!info) {} as it is an error..

What is the reason for $url = $r->get( 'url' ); to fetch the file address with the render server local IP? How can I fix it? And how can the Download button on mediawiki triggers a "file not found" from computers on the local network even though the $url is correct?

I was able to resolve this by changing the port for nslave

mw-qserve&nserve --port 8080&nslave --cachedir ~/cache/ --serve-files-port 80&postman&

Hello,

Is there a switch to force Collection to follow page redirects when rendering books? As of now it only prints #REDIRECT [[Target page]].

Thanks!

I don't know about the software, but the best thing to do is to is to edit the Book page and correct the link, using[[Correct destination page|Chapter title]]. ~~~~

Yes, that is what i'm doing at the moment. But the books we use have hundreds of pages and many of those pages often change names.

Hi all,

Hopefully someone can guide me on this topic. I recently installed the Collection extension in our MW 1.28 environment but I am seeing some issues when exporting pages which contain foreign characters as PDFs. Have a look:

This:

回答が遅くなっておりまして申し訳ありません。 1つ目のご質問について、現在調査中のため、回答までお時間をいただけます ようお願いいたします。

Turns into this:

https : //ibb.co/WGBxdqs

I've read through the extension doc as well as the mwlib doc but I've been unable to resolve the issue. Is there something else I'm missing, a font, a setting, anything?

Thanks!

OK, so this definitely appears to be a missing font issues...however when I installed missing fonts and symlinked the issue remains.

ln -s /usr/share/fonts/truetype/* /mwlibfonts/

Are any additional edits required for fontconfig.py? github.com/pediapress/mwlib.rl/blob/master/mwlib/rl/fontconfig.py

Any help greatly appreciated!

figured it out... /mwlibfonts/ was at the root not ~/mwlibfonts/ moving the font symlinks to the root user's home fixed it.

Now for unicode support...

🦄 = ᾘ?

Any thoughts on an additional font package required to handle this?

AFAIK, emojis have never worked so far. If you want to dive into that, you can either add an Emoji font like Symbola (Download) but this will only give you only black and white Emojis.

For color Emojis, you could look into a package like Emojipy or Emoji to bring Emoji capabilities to Reportlab. This six year Blogpost outlines the approach.

Thank you for the pointers, I'll have a look.

Hello @all,

i have fresh installed Mediawiki 1.35.1 and the lates Collection Version.

I can start the bookgenerator but the i get the Messages:

Der Buchgenerator wird Änderungen unterzogen

Aufgrund mehrerer Probleme mit unserem vorhandenen System unterstützt der Buchgenerator das Speichern von Büchern als PDF-Datei nicht mehr.

The book generator is undergoing changes

Due to several problems with our existing system, the book generator no longer supports saving books as PDF files.

What can i do?

Many thanks in advances

TBC

The same error is on that wiki here if you would like to create a book!

see Topic:Vz3dmiglljmmfspo

wikitext markup should be removed in the printed book =>

https://pediapress.com/books/setup/8934ced7cd88c0ebfa5f925249d425/

Christian 🇫🇷 FR (talk) 22:49, 9 December 2020 (UTC)

Tracked in Phabricator
Task T256351

For private wiki's the Extension:Collection doesn't work

$wgGroupPermissions['*']['read'] = false;

Is there any solution?

Here the readapidenied errors from mwlib:

   creating nuwiki in u'/var/cache/mwlib/7e/7e607dca372f1977/tmpXYsGQC/nuwiki'

   ERR: You need read permission to use this module.: [fetching https://<domain_name>/wiki/api.php?action=query&meta=siteinfo&siprop=general|namespaces|interwikimap|namespacealiases|magicwords|rightsinfo&format= json]

   ERR: You need read permission to use this module.: [fetching https://<domain_name>/wiki/api.php?action=query&meta=siteinfo&siprop=general|namespaces|interwikimap|namespacealiases|magicwords&format=json]

   ERR: You need read permission to use this module.: [fetching https://<domain_name>/wiki/api.php?action=query&meta=siteinfo&siprop=general|namespaces|interwikimap|namespacealiases&format=json]

   ERR: You need read permission to use this module.: [fetching https://<domain_name>/wiki/api.php?action=query&meta=siteinfo&siprop=general|namespaces|interwikimap&format=json]

   removing tmpdir u'/var/cache/mwlib/7e/7e607dca372f1977/tmpXYsGQC'

   memory used: res=33.8 virt=230.7

   1% error Traceback (most recent call last):

     File "/usr/local/bin/mw-zip", line 11, in <module>

       sys.exit(main())

     File "/usr/local/lib/python2.7/dist-packages/mwlib/apps/buildzip.py", line 155, in main

       make_zip(output, options, env.metabook, podclient=podclient, status=status)

     File "/usr/local/lib/python2.7/dist-packages/mwlib/apps/buildzip.py", line 50, in make_zip

       make_nuwiki(fsdir, metabook=metabook, options=options, podclient=podclient, status=status)

     File "/usr/local/lib/python2.7/dist-packages/mwlib/apps/make_nuwiki.py", line 192, in make_nuwiki

       pool.join(raise_error=True)

     File "/usr/local/lib/python2.7/dist-packages/gevent/pool.py", line 433, in join

       greenlet._raise_exception()

     File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception

     File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run

     File "/usr/local/lib/python2.7/dist-packages/mwlib/apps/make_nuwiki.py", line 119, in run

       self.fetch_pages_from_metabook(api)

     File "/usr/local/lib/python2.7/dist-packages/mwlib/apps/make_nuwiki.py", line 59, in fetch_pages_from_metabook

       fetch_images=not self.options.noimages)

     File "/usr/local/lib/python2.7/dist-packages/mwlib/net/fetch.py", line 282, in __init__

       siteinfo = self.get_siteinfo_for(self.api)

     File "/usr/local/lib/python2.7/dist-packages/mwlib/net/fetch.py", line 453, in get_siteinfo_for

       return m.get_siteinfo()

     File "/usr/local/lib/python2.7/dist-packages/mwlib/net/sapi.py", line 183, in get_siteinfo

       raise RuntimeError("could not get siteinfo")

   RuntimeError: could not get siteinfo

    in function system, file /usr/local/lib/python2.7/dist-packages/mwlib/nslave.py, line 64

The error indicates that your API is not accessible for the renderer. The renderer must be able to fetch data via the MW API in order to render the content. I can't give you any specific tips on how to resolve this, unfortunately.

Actually T245149 addresses a similar issue in 1.34, which should be fixed in 1.34.1 (see Release notes/1.34#MediaWiki 1.34.1)

Our wiki is running on MediaWiki 1.31.7 and using Collection 1.7.0 (af3a0b8) 14:23, 15 April 2018. The Download as PDF is constantly failing and directing the user to Reading/Web/PDF Functionality which doesn't specifically address the reason for the "Book rendering failed". Reading through Talk:Reading/Web/PDF Functionality doesn't clear up the situation much either. It does seem to indicate there is a new render server available at https://pediapress.com/collector but that doesn't seem to work for non-Wikipedia sites. The existing render server https://tools.pediapress.com/mw-serve/ does seem to still active.

Is the functionality via this extension dead for low traffic sites that don't need or cannot install (i.e shared hosting) their own PDF server?

The pyfribidi package (of mwlib) must be installed to render right-to-left texts.

Hi,

Extension:Collection is installed on MW1.31 and is running on its own rendering server; mwlib runs as a standalone server. But the rendering of the arabic text is left-to-right only. Is there a known bug or do I miss something in the configuration?

Thank you in advance!

There is a discussion here which is suggesting that the Wikipedia Books function be withdrawn. The RfC concerns user interface hooks to the collection extension and the discussion has broadened into the whole Books namespace and this software. You are invited to contribute.

I understand that PediaPress are, or were, heavily involved in maintaining this extension. Anybody who knows their full relationship with the WMF would also be more than welcome to give a clarification, as the upload link to their PoD service is also being criticised.

Thank you for the heads up. I will try to clarify as much as I can.

@Steelpillow, thanks for vouching for the Collection extension. It's a valuable tool for the wiki that I'm running (http://wikimedi.ca). Once we have the resources, we'll try and figure out a solution to get it back up and running.

We're anxiously waiting for a new version of the Collection extension and we will happily devote some ressources maintaining and improving it .

@Tinss: The book rendering service is new, the collection extension will need only small updates to link to it.

User:Ckepper at PediaPress is writing a new PDF renderer. I do not know if he will accept help, but you can ask. (His user page here seems very out of date).

User:Dirk Hünniger maintains the MediaWiki2LaTeX renderer. It is available but needs some work, both on the software, which is written in an unusual language called Haskell, and on the service configuration. I think he will be glad of some proper help.

I have 2 environments that I (hope) think are identical. They are both setup with:

MW 1.25alpha PHP 5.3.17 Collection 1.7.0 (1618025)

In the Dev environment, Collection works perfectly. In the Prod environment, I'm seeing this error:

<Greenlet at 0xd43eb0: <bound method start_fetcher.run of <mwlib.apps.make_nuwiki.start_fetcher object at 0xdbba50>>> failed with  RuntimeError

   creating nuwiki in u'/u1/wiki_pdf/cache/2e/2e58aa5a7230f6a6/tmpJuZHDI/nuwiki'
   ERR: <urlopen error [Errno 111] Connection refused>
   ERR: <urlopen error [Errno 111] Connection refused>
   ERR: <urlopen error [Errno 111] Connection refused>
   ERR: <urlopen error [Errno 111] Connection refused>
   removing tmpdir u'/u1/wiki_pdf/cache/2e/2e58aa5a7230f6a6/tmpJuZHDI'
   memory used: res=18.2 virt=152.7
   1% error Traceback (most recent call last):
     File "/usr/local/bin/mw-zip", line 9, in <module>
       load_entry_point('mwlib==0.15.14', 'console_scripts', 'mw-zip')()
     File "/usr/local/lib64/python2.6/site-packages/mwlib/apps/buildzip.py", line 155, in main
       make_zip(output, options, env.metabook, podclient=podclient, status=status)
     File "/usr/local/lib64/python2.6/site-packages/mwlib/apps/buildzip.py", line 50, in make_zip
       make_nuwiki(fsdir, metabook=metabook, options=options, podclient=podclient, status=status)
     File "/usr/local/lib64/python2.6/site-packages/mwlib/apps/make_nuwiki.py", line 192, in make_nuwiki
       pool.join(raise_error=True)
     File "/usr/local/lib64/python2.6/site-packages/gevent/pool.py", line 98, in join
       raise greenlet.exception
   RuntimeError: could not get siteinfo
    in function system, file /usr/local/lib64/python2.6/site-packages/mwlib/nslave.py, line 64

So it's failing on the start_fetcher.run method, but I can't seem to figure out why. Is this likely some system configuration thing?

Solved.

This was a DNS issue. We had to add and entry to /etc/hosts so that the server could talk to itself, basically.

Hi, I am getting the same error. Do you happen to know what you put in your /etc/hosts file to resolve this?

I am getting same error after enabling SSL. Could you share how did you resolve this issue

Yes, please, show us what you did to your /etc/hosts file. I need help with that, too.