Topic on Project:Support desk

Hey. I keep getting this error button and I don't know how to fix it. It can be seen here: http://dragon-mania-legends-wiki.mobga.me/Ruins, here: http://dragon-mania-legends-wiki.mobga.me/Habitats and here: http://dragon-mania-legends-wiki.mobga.me/Category:Images:Calendar_Events

I've tried setting a temp folder ($wgTmpDirectory = "$IP/images/temp";) and a debug file ($wgDebugLogFile = "/var/log/mediawiki/debug-{$wgDBname}.log";) as suggested and even made the temp folder writable by the server using chown, but that did not work.

The manual on how to debug specifies that this log can contain sensibile information, thus I won't post all of its contents and would like to know what could I share here from it that would help in solving this issue?

I mention the console is displaying some weird errors, but I cannot interpret them, sadly.

Looking at your Special:NewFiles, it looks like you are able to upload images without problem, but those particular images cause a problem.

I can use thumb.php to create thumbnails of various sizes (see also).

thumb.php appears to work on images that show the problem.

Meanwhile, fails to produce thumb.php some thumbnails for the problematic images.

Maybe you can see what is produced in the debug log when one of the failing thumb.php links is called?

I found no recent instance of the cupcake thumb (at least the date isn't displayed when such calls are made, so it might be recent), thus I also included something more recent. The cupcake thumb is at the very top, the others are separated by a large space: http://pastebin.com/Yg6V1Zr6 Hope it helps!

I'm at a loss. I don't really know how to help you without access to your machine. Sorry. Perhaps you can check the permissions on all the directories in your images folder to see if one or two has funky permissions.

What if you set $wgShowExceptionDetails to true temporarily and open the problematic thumb.php URL?

That string was already set to true even before the problem occurred. But here's the strange thing. If you go and check my first two links (ruins and habitats), you'll see that the problem was solved on its own...

Here's a screenshot of the permissions that all folders seem to have: http://prnt.sc/em8bhd I selected all of them, and made sure they are all set to that. Do I need to change anything?

I forgot to mention that when renaming the file, the issue seems to disappear (of course, there's a reason I named them the way I did, in the first place, thus this isn't a good solution or the solution for that matter). And, in some cases, the issue is triggered only if I get over a certain pixel size while adding it to the article. Would manually moving the images (on the server) from one folder to another solve it? Is it even safe to do that?

The permissions look fine if that user is the user that runs as the web server. You could also change the permissions on images to the numeric code 777 and recurse through the images directory.

You could move the images directory, but you would need to change a few configuration options like Manual:$wgUploadPath or Manual:$wgUploadPath. You might also need to use Manual:Img auth.php to stream images.

I am not sure if that user is the one that runs as the web server.. I was using the root account. How do I find out who it is?

Also, the second option you gave me is the same as the first one. I was asking if I could only move the images that cause trouble, not the whole directory. Seems kinda pointless to move the whole thing just for a few troublemakers.

• I don't recognize the tool you are using for ftp. But what does the full listing of the mediawiki directory show? Maybe that would have user names on it.
• If you can physically remove the primary uploaded problematic file, then I think you could re-upload it. That might solve the problem.

• I use FileZilla Client. I am not sure what you're referring to. Is this it:http://prnt.sc/emeu0n ?
• By physically removing it, you mean to manually delete it from the server itself or from the wiki? Because if it's the latter, it doesn't work.

• That has the information I wanted. It looks like you're on a Debian server and the permissions are set correctly.
• I meant from the server itself.
• Could you use FileZilla to get the listing of the images/thumb/9/90 and images/thumb/9/90/The_Winds_of_November_Event_%2816.11.14%29.jpg directory?

I managed to get there and the folder has the image in 3 different sizes: 200px, 300px and 400px. Interestingly enough, this folder seems to have another owner than the typical one: http://prnt.sc/emrhbx I checked other problematic files, thinking the owner might be at fault, but they had the typical owner, so I guess the different owner theory can (for now, at least) be ruled out.

Should I delete the image from both its permanent place and the thumb folder?

The root owner is definitely going to cause a problem. Delete that directory and its contents.

That solved the thumbnail problem, but only for that image.

Ehm, I was thinking of something. Sometimes, some special pages (usually uncategorised files) won't update, no matter what, regardless if I run updateSpecialPages.php (which runs daily on a cron job) no matter what. So I force it with refreshLinks.php which, in turn, creates a ton of jobs which are run by runJobs.php later on. From time to time, some jobs just get stuck, not allowing themselves to be run. I posted about it 2 months ago (https://www.mediawiki.org/wiki/Topic:Tjumyamw7pqzvjr1) but got no response. So I looked into the database and found the stuck jobs in the job table. So I just manually deleted them (and have been doing that ever since). Could this be the problem behind it or is it unrelated? I remember this particular issue happened on the 1.26.2 installation, then a failed server upgrade brought the wiki down for a little while, we moved servers and that issue was fixed (this was last summer).

It could be related.

You need to make sure that runJobs.php and refreshLinks.php are running as the www-data user and not the root user.

I am sorry, I am not good at this. How do I do that? I haven't used any other user besides my personal one and the root one.

How do you normally get the jobs to run? How are they being run right now?

With cron. When I set it up, I opened putty, logged in on root, entered the command: crontab -e then I added this string: */120 * * * * cd (path to wiki) && php maintenance/runJobs.php and saved. But after using refreshLinks, I run it manually. It usually doubles the number of jobs, initially, then starts to decrease them.

If the only cron jobs you have are those that are running for the wiki, then run this sequence of commands as root:

# crontab -l > cron.txt         # to save the current crontab
# crontab -r                    # to delete the current crontab for root
# crontab -u www-data cron.txt  # to give the crontab entries to the www-data user.

If you have other crontab entries, skip the second step (which removes all the entries for root) and edit cron.txt so that only wiki maintenance scripts are left in the file before running the third command.

In any case, the first command will back up all your root cron commands if there is a mistake made.

Before proceeding, I used grep CRON /var/log/syslog to see the other jobs. There are 3 other that are not in my current crontab (when using crontab -e), but they all are run by root. Two seem to be more server-related: http://prnt.sc/emzniy

Is it safe to still do all the steps?

Also, I did the first step, but I got no confirmation of whether the procedure was successful or not? How do I check?

Ah, also, deleting images from the server then re-uploading them doesn't get rid of the error (because it uploads them to the same place in which they were before), BUT, I think it might just be related to root being the owner. So far, all images that I checked in images/thumb that had the issue, had root as the owner. It seems this issue occurs only with thumbnails.

You can see if the first step worked by looking for cron.txt in the directory where you ran the file.

In fact, since you have other cron jobs running as root, skip the second step and just edit the cron.txt file to have the wiki-related jobs.

After that, erase those jobs from the root crontab using crontab -e and then run the third command to move the cronjobs to the www-data user.

I am not exactly sure what I am doing wrong. Technically, I run the command in the base directory (that is, I didn't change the folder after opening putty), but there is no such file there. Then I looked in /var/spool/cron/crontabs, which has the crontab that I can access through the crontab -e and crontab -l commands, but that's the only file in there (googling also lead me to the same location). There's also another crontab in /etc which lists all the cronjobs. But no sign of cron.txt in any of those three spots....

If you run "crontab -u root -l > cron.txt" as root, it should create the cron.txt file for you.

I did that, still wasn't in any of those three locations, but I used find / -name and found it in /root. Inside it, there are the two jobs: runJobs.php and updateSpecialPages.php and the default "# Edit this file to introduce tasks to be run by cron". Is this the right file?

That is it. It makes sense that it is in /root since that is the root user's home directory. Probably typing ls after running the command would have shown it.

Awesome. So let me get this straight before doing anything. I need to edit that file to have the wiki-related jobs going. It already has two jobs (updateSpecialPages and runJobs). This means I need to add the other three remaining jobs from /etc/crontab?

These are:
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

Just the first one. And "root" should not be there. You should probably paste your cron.txt before running the command to load it.

Is this correct: http://pastebin.com/2QGi7Wjf What do I do with the last 3 cronjobs and with the other two crontabs (from /var/spool and /etc)?

EDIT: There was another job that I missed which was added as custom, apparently:

# OJG CUSTOM JOBS ADDED FOR THE WIKI:
*/15 * * * *	root	cd /(path to wiki)/ && /usr/bin/php maintenance/rebuildFileCache.php >> /var/log/(log folder)/rebuildFileCache.log 2>&1
5 2 * * *     root    cd /(path to wiki)/ && /usr/bin/php maintenance/runJobs.php >> /var/log/(log folder)/runJobs.log 2>&1

It seems there is a second runJobs which is a tad bit different than the first runJobs. Doesn't really seem to be doing its job properly, though. Should I just remove it?

First, do not run crontab -r. Use crontab -e to edit the root crontab files and remove only the wiki crontab entries. The rest should remain.

You should have a file, cron.txt, with the following based on the information you've given here:

# OJG CUSTOM JOBS ADDED FOR THE WIKI:
*/15 * * * * www-data  cd (path to wiki) && php maintenance/rebuildFileCache.php   >> (log folder)/rebuildFileCache.log 2>&1
*/15 * * * * www-data  cd (path to wiki) && php maintenance/runJobs.php            >> (log folder)/runJobs.log 2>&1
0    0 * * * www-data  cd (path to wiki) && php maintenance/updateSpecialPages.php >> (log folder)/updateSpecialPages.log 2>&1

Move the cron.txt file to /etc/cron.d/wiki-cron-jobs.

This will allow Ubuntu's cron system to run them. Anytime you need to change them, you can safely edit that file.

I think this will get you where you need to be. Sorry for the confusion.

And to reduce the repetition, you can use shell variables in that file, so you could do the following:

LOGDIR=/var/log/(log folder)
WIKIDIR=(path to wiki)

# OJG CUSTOM JOBS ADDED FOR THE WIKI:
*/15 * * * * www-data  cd $WIKIDIR && php maintenance/rebuildFileCache.php   >> $LOGDIR/rebuildFileCache.log 2>&1
*/15 * * * * www-data  cd $WIKIDIR && php maintenance/runJobs.php            >> $LOGDIR/runJobs.log 2>&1
0    0 * * * www-data  cd $WIKIDIR && php maintenance/updateSpecialPages.php >> $LOGDIR/updateSpecialPages.log 2>&1

I did that and moved the .txt to where you suggested (I had to manually create the wiki-cron-jobs folder). Here's the entire paste: http://pastebin.com/jY6sfM5f

Now, one question. What do I do with the remaining cronjobs that I wrote here (https://www.mediawiki.org/w/index.php?title=Topic:Tn0u0v07qa9cb9v8&topic_showPostId=tn9kgkwpt0b8dob9#flow-post-tn9kgkwpt0b8dob9)? They weren't in the crontab file that I could open with crontab -e while logged in on root. They are in the crontab that's found in /etc. This is its contents: http://pastebin.com/vDZtEWML

I'm sorry, I should have been clearer. I wanted you to move cron.txt to a file called /etc/cron.d/wiki-cron-jobs. If you've created a directory, I'm not sure it will work.

Remove these lines from the file you found under /etc:

# OJG CUSTOM JOBS ADDED FOR THE WIKI:
*/15 * * * *    root    cd /(wiki path) && /usr/bin/php maintenance/rebuildFileCache.php >> /var/log/wiki_logs/rebuildFileCache.log 2>&1
5 2 * * *     root    cd /(wiki path) && /usr/bin/php maintenance/runJobs.php >> /var/log/wiki_logs/runJobs.log 2>&1

The others should be left as-is.

Well, there was no such file there. Cron.txt currently rests in /etc/cron.d/wiki-cron-jobs folder. I used grep CRON /var/log/syslog to see the recent jobs, but not jobs from cron.txt seem to have been executed... what should I do?

Run the following commands:

$ mv /etc/cron.d/wiki-cron-jobs/cron.txt /etc/cron.d/wiki-jobs
$ rmdir /etc/cron.d/wiki-cron-jobs

Both should run without problem and you should see cron jobs executing shortly after that.

Hey. Thanks! But there seems to be a problem... the jobs aren't run. Using grep CRON /var/log/syslog it does show me the runJobs.php has been executed, but the number does not go down, unless I manually run it. What might be the cause?

Actually, it seems none of the maintenance scripts are working, not only php runJobs.php.

Could you run the following and tell me the output:

$ grep wiki-jobs /var/log/syslog

This is what I get: http://prnt.sc/eo1iyo

So, it is loading properly. I was worried that I had given you a bad one or that it had been corrupted some way. For example, I created a file with the wrong owner and got the following:

$ grep test /var/log/syslog
Mar 24 12:17:01 web cron[369]: (*system*test) WRONG FILE OWNER (/etc/cron.d/test)
$

I have set up a wiki with a file similar to what I asked you to set up now and when I look for www-data in my syslog, I see the following:

$ grep www-data /var/log/syslog
...
Mar 24 12:30:02 web CRON[25865]: (www-data) CMD ( cd $WIKIDIR && php maintenance/rebuildFileCache.php   >> $LOGDIR/rebuildFileCache.log 2>&1)
Mar 24 12:30:02 web CRON[25867]: (www-data) CMD ( cd $WIKIDIR && php maintenance/runJobs.php            >> $LOGDIR/runJobs.log 2>&1)
...

Do you see anything similar?

It seems I do: http://prnt.sc/eo7t6b

If the command seems to be executed correctly, it means that wikidir and logdir are set correctly, right? I am not sure if it's safe to post them here (that's why I always censor them), but I wrote them something like:

LOGDIR=/var/log/foldername/ WIKIDIR=/mainfolder/foldername/

I didn't add /maintenance/ at the end of wikidir since the cronjob automatically switches to the maintenance folder due to php/maintenance. Plus, the command is basically the same to how I had it in the root crontab (minus adding information to the log, as well as having the name of the wiki path written instead of wikidir).

I take it that the above steps are correct?

Yep, it looks like all the steps are correct. Check your log files to see what is being put in them.

From the outside, when looking at 'last modified' it seems that rebuildFileCache.log has been last modified on the 23rd while runJobs.log on the 18th. Those are the only two logs inside there.... The owner/group is root root. The first log is huge, I copied pasted the last pages (132 in word) and replaced the path to the wiki with 'wikidir':http://pastebin.com/quGJUa50

As for runJobs = http://pastebin.com/rWejzRu7

So, for rebuildFileCache, it looks like it is failing. As the message says, "Set $wgShowExceptionDetails = true; and $wgShowDBErrorBacktrace = true; at the bottom of LocalSettings.php to show detailed debugging information." so we can see what is going on.

runJobs looks fine.

oh, and make both log files owned by www-data so that the cron jobs will be able to update them. You should also change the ownership on the log directory.

Both file logs and the folder itself are owned by www-data now. I am not sure how runJobs looks fine since I can see that the jobs aren't being run.

Maybe the wiki-jobs file itself has to be owned by www-data? Most of the folders on the server are owned by root.

Off-topic: most of the wiki folder is owned by www-data, but some folders are owned by root. Should I pass ownership to www-data?

And this is what rebuildcache log shows now: http://pastebin.com/SPeztVnt

You still haven't set the variables to show exceptions.

I actually did. This is what the bottom of localsettings looks like: http://pastebin.com/Sej3nm9g

Also, it's owned by root.

interesting... it doesn't seem to have made a difference for your rebuildcachelog.

Maybe I should give those files different permissions? Their current permissions are -rw-r--r--

Also, what should I do to get the runJobs script running? Or all of the scripts, for that matter. None seems to be working.

Here are the permissions on my file:

$ ls -l /etc/cron.d/test 
-rw-r--r-- 1 root root 442 Mar 24 12:15 /etc/cron.d/test
$

The content is what I gave above.

My log files show that the cron jobs are running.

$ zgrep maintenance /var/log/syslog.1 | tail -53 | head -3
Mar 26 00:00:01 web CRON[20974]: (www-data) CMD ( cd $WIKIDIR && php maintenance/rebuildFileCache.php   >> $LOGDIR/rebuildFileCache.log 2>&1)
Mar 26 00:00:01 web CRON[20977]: (www-data) CMD ( cd $WIKIDIR && php maintenance/runJobs.php            >> $LOGDIR/runJobs.log 2>&1)
Mar 26 00:00:01 web CRON[20979]: (www-data) CMD ( cd $WIKIDIR && php maintenance/updateSpecialPages.php >> $LOGDIR/updateSpecialPages.log 2>&1)
$

I'm not sure how I can help you in this forum.

So I fixed it. All I did was to replace wikidir and logdir with their respective folders. Strange fix.

Now, all that remains is the stuck jobs. Do I manually remove them from the database or what should I do? So far, that's the only solution I found to get rid of them.

If the jobs are running and the stuck ones are not clearing, then, yeah, I think that is what you have to do.

Hrm... I am thinking I should have put export in the cron file. *sigh*...

Then I guess this settles it, finally.

Thank you very much for all the help and the patience! Much appreciated.

Off-topic: is there a MediaWiki-recommended list of permissions and owners/groups that folders on a server should have?

We can certainly start creating such a list. Seems like something that should be in an FAQ.

That would be great. Thanks for now! Your help is really appreciated. Going to mark this as resolved now.

Hey. I need to reopen the topic because I am getting this error again. So far:
It isn't caused by any image being under the ownership of root as all are under www-data.
The cronjobs are all ran by www-data
The issue seems to appear after rebuildFileCache.log is ran and pages that get this error (example: http://dragon-mania-legends-wiki.mobga.me/Dragon_League and http://dragon-mania-legends-wiki.mobga.me/Decorations) can simply be fixed with a null edit. Bonus: it affects Comments too, even though I added the lame-workaround that I was suggested in phabricator:T152764 (not in the shell, but the "Invalid tag: comments" message will appear wherever the tag is).
The log doesn't seem to show anything off:https://pastebin.com/U1Ck1hcD (though I've extracted it before turning debugging of.. should I wait for it to be ran again after turning debugging on?)

Anyone any ideas?

After turning on everything, the runJob and rebuildfilecache log still didn't give me anything out of the ordinary, but the debugging log file did: https://pastebin.com/7xLyf2A5

Anything seems to be off there?

I don't see any error in the debug log either. So are you still seeing "Error creating thumbnail: Unable to save thumbnail to destination"? that would be pretty strange

About the comments extension, it's normal that it doesn't display the extension, or it displays the invalid tag, because the workaround is disabling the extension on command line (which is what generates the file cache).

The thing is that it also fixes on its own from time to time, but I am just not able to figure what causes it in the first place and what fixes it after. Aside from those three maintenance scripts that are run by www-data, I also have these two other scripts run by root (http://prnt.sc/ey9ns3) which don't seem to have to do with the wiki folder itself. Could they have any role in all this?

As I previously said, a simple null edit will fix it, so I don't know. The error just pops here and there on at least those two pages I have linked above, then it just goes away on its own.

Also, I don't know if they are related or not, but I get these two notices on this page only (at least it's only here that I saw them): http://dragon-mania-legends-wiki.mobga.me/Special:ListGroupRights

Not related. This may have something to do with a weird setting of $wgAddGroups/$wgRemoveGroups/$wgGroupsAddToSelf/$wgGroupsRemoveFromSelf

Does it affect the wiki with anything? If not, it just goes away when exception details and show backtrace are displayed and there's really no point in bothering too much over such a small thing (at least for now).

Any idea about the thumbnail error?