Topic on Project:Support desk

Problem with installation of Extension:CheckUser reported in task T155163

Hi, For some reason the Special:RecentChanges stopped working after installing 1.28.0. It gets updated after running from the command line 'rebuildall.php' but than it stops updating again till you run again this command. The variable $wgMiserMode isn't set in LocalSettings.php (so is default (false). Setting it to false doesn't affect the problem. Running 1.28.0 with PHP 5.6.29 (fpm-fcgi) and MySQL 5.6.23-log. Thanks, --Albert Ke (talk)
Update: Users contributions do get updated, just the recent changes that doesn't. Any hint? --Albert Ke (talk)

This may be a problem with your job queue. Are you changing how it is run? See Manual:Job queue for more info.

Thank you MarkAHershberger, I'm normally running the ./runJobs.php once in a while till it is done when e.g. updating MSW tables. So nothing set in the LocalSettings.php. It seems like the Job queue doesn't finish anymore though (can run it for hours) and it has id numbers that are way larger than every before (1157954).

Can you set it to run automatically? Instead of relying on you to manually purge it?

I just set it up as a cron for after midnight. I'll post in a day or so if it worked. --Albert Ke (talk)

Hi, on top of having set up a cron for runJobs.php, also ran it to empty queue. Got from 30,000 to ~1,400 but can't seem to empty the queue further. I get a few errors once in a while:

1)
PHP Catchable fatal error: Argument 1 passed to RefreshLinksJob::runForTitle() must be an instance of Title, null given, called in .../includes/jobqueue/jobs/RefreshLinksJob.php on line 117 and defined in .../includes/jobqueue/jobs/RefreshLinksJob.php on line 131

2)
[6e1f392a9e9adc8491f78a47] [no req] JobQueueError from line 813 of .../includes/jobqueue/JobQueueDB.php: DBQueryError: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading?
Query: UPDATE `mw_job` SET job_token = ,job_token_timestamp = '20170108230740' WHERE job_id IN ('1857267','1880624','1947415','1961303','1950761') Function: JobQueueDB::recycleAndDeleteStaleJobs Error: 1205 Lock wait timeout exceeded; try restarting transaction (localhost)

Backtrace:

1. 0 .../includes/jobqueue/JobQueueDB.php(706): JobQueueDB->throwDBException(DBQueryError)
   
2. 1 .../includes/jobqueue/JobQueueDB.php(301): JobQueueDB->recycleAndDeleteStaleJobs()
   
3. 2 .../includes/jobqueue/JobQueue.php(372): JobQueueDB->doPop()
   
4. 3 .../includes/jobqueue/JobQueueGroup.php(240): JobQueue->pop()
   
5. 4 .../includes/jobqueue/JobRunner.php(159): JobQueueGroup->pop(integer, integer, array)
   
6. 5 .../maintenance/runJobs.php(87): JobRunner->run(array)
   
7. 6 .../maintenance/doMaintenance.php(111): RunJobs->execute()
   
8. 7 .../maintenance/runJobs.php(119): require_once(string)
   
9. 8 {main}
   

3) (other functionality that stopped)
Special:Version page stopped working now as well. Gives the following error when loading: csdms.colorado.edu is currently unable to handle this request.

4)
Mediawiki administrative page, under "Data repair and upgrade" doesn't update any further than 80%.

Additionally I turned off the Widgets for now as it gave the following warning when running the runJobs.php: PHP Warning: touch(): Utime failed: Permission denied in .../extensions/Widgets/smarty/libs/sysplugins/smarty_template_compiled.php on line 195

Also removed SemanticMediawiki Result Formats as I think it caused the following PHP Fatal error: Allowed memory size of 188743680 bytes exhausted (tried to allocate 1176309 bytes) in .../extensions/SemanticMediaWiki/includes/SMW_Outputs.php on line 163

Is there another way to empty the Jobs queue, and would that solve the problem? Thanks, --Albert Ke (talk)

It looks like you haven't run update.php. Could you run that and then try again?

I did ran it multiple times but that error message keeps coming up. The script runs fine in 0.1 s.
Turned off SMW for now, ran the update.php and after that the runJobs.php. It looks like the runJobs.php sometimes errors out on old namespaces that are removed but ran all the way till it doesn't do anymore on the command ./runJobs.php. Did get these errors:

2017-01-09 18:33:56 refreshLinks Template:Attribute_movie1 pages={"3094":[200,"TidalBoreAK"]} rootJobSignature=627bbcccccdfc74b05a5efa71ecec4bde4b08404 rootJobTimestamp=20170109160845 triggeredRecursive=1 requestId=6962d188e12b48fe8933016b (id=2308894,timestamp=20170109163310) STARTING PHP Catchable fatal error: Argument 1 passed to RefreshLinksJob::runForTitle() must be an instance of Title, null given, called in .../includes/jobqueue/jobs/RefreshLinksJob.php on line 117 and defined in .../includes/jobqueue/jobs/RefreshLinksJob.php on line 131

Data of the old namespaces was copied to another namespace and removed afterwards. Looks like some template still hold a link to these old namespaces that are removed ... 2-3years ago.

Although php ./runJobs.php doesn't return anything when run anymore (so queue empty?), there are still jobs in the queue: http://csdms.colorado.edu/mediawiki/api.php?action=query&meta=siteinfo&siprop=statistics

And the RecentChanges page doesn't update either: http://csdms.colorado.edu/wiki/Special:RecentChanges and http://csdms.colorado.edu/wiki/Special:Contributions/WikiSysop to show that more changes are made to the wiki then the RecentChanges indicates. And http://csdms.colorado.edu/wiki/Special:Version won't show anymore.

I did ran php namespaceDupes.php --fix (3 fixes) and refreshLinks.php but that didn't do the trick.

Thanks for helping out!! --Albert Ke (talk)

RecentChanges display changes except the most recent ones from your account (and maybe others). Looks like running runJobs.php makes it to update. Setting Manual:$wgRunJobsAsync to false should fix this issue.

You can take a look at the job table and maybe delete them if they're really old (look at the job_timestamp field)

Hi Ciencia, RecentChanges does only get updated when I run ./rebuildrecentchanges.php. That is why you see some changes made today and from a few days back. RecentChanges stopped monitoring changes automatically though when I installed 1.28.0 some days ago. I did set the $wgRunJobsAsync to false now. Did an update.php and runJobs.php, no changes. I can start edit the job table, haven't done so. Would you happen to know a link on how best to manually truncate jobs in mediawiki? Thanks, --Albert Ke (talk) 22:29, 9 January 2017 (UTC).

Ok., I think I figured it out how to truncate the job table in MW but doesn't look like that is helping me. This is what I did after login in to the MW mysql table:

truncate table job;

Query OK, 0 rows affected (0.13 sec)

When looking at the job table through: php ./showJobs.php --group it gave me however:

refreshLinks: 0 queued; 720 claimed (0 active, 720 abandoned); 0 delayed

htmlCacheUpdate: 0 queued; 2 claimed (0 active, 2 abandoned); 0 delayed

refreshLinksPrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed

refreshLinksDynamic: 0 queued; 11 claimed (11 active, 0 abandoned); 0 delayed

cirrusSearchLinksUpdate: 0 queued; 44 claimed (0 active, 44 abandoned); 0 delayed

cirrusSearchLinksUpdatePrioritized: 0 queued; 7 claimed (6 active, 1 abandoned); 0 delayed

So still abandoned jobs in the queue.... Anyway to remove those (if that is affecting the Special:RecentChanges (not updating) and the Special:Version (not showing) at all. Thanks!, --Albert Ke (talk) 22:50, 9 January 2017 (UTC)

Are you sure you truncated the job table of the right database? No jobs should appear if you really truncated it.

Hi Ciencia, Would you be able to point out what commands to use otherwise? I have indicated what I used above and pretty sure I used the mw database. Thank you!, --Albert Ke (talk) 15:40, 10 January 2017 (UTC)

1. truncate table job; ← this should leave the table empty.
2. select * from job; ← should give 0 rows.
3. http://csdms.colorado.edu/mediawiki/api.php?action=query&meta=siteinfo&siprop=statistics ← should give 0 jobs.

If #3 doesn't show 0 jobs (unless someone has edited while executing it), you're obviously running the commands on the wrong database.

Thank you Ciencia,
I followed your suggestion. Used the user, password and database name defined in the LocalSettings.php to use the right database. Executed the commands you indicated:

mysql> show tables;

+--------------------------------+

| Tables_in_csdms_wiki |

+--------------------------------+

| archive |

| category |

....

| job |

....

184 rows in set (0.00 sec)

mysql> truncate table job;

Query OK, 0 rows affected (0.04 sec)

mysql> select * from job;

Empty set (0.00 sec)

Went to the api; 4 jobs in queue. So not completely empty. So for some reason those 4 jobs did not get removed. This is the database for the wiki though.....

mysql> show columns from csdms_wiki.job;

+---------------------+------------------+------+-----+---------+----------------+

| Field | Type | Null | Key | Default | Extra |

+---------------------+------------------+------+-----+---------+----------------+

| job_id | int(10) unsigned | NO | PRI | NULL | auto_increment |

| job_cmd | varbinary(60) | NO | MUL | | |

| job_namespace | int(11) | NO | | NULL | |

| job_title | varbinary(255) | NO | | NULL | |

| job_timestamp | varbinary(14) | YES | MUL | NULL | |

| job_params | blob | NO | | NULL | |

| job_random | int(10) unsigned | NO | | 0 | |

| job_attempts | int(10) unsigned | NO | | 0 | |

| job_token | varbinary(32) | NO | | | |

| job_token_timestamp | varbinary(14) | YES | | NULL | |

| job_sha1 | varbinary(32) | NO | MUL | | |

+---------------------+------------------+------+-----+---------+----------------+

Exit database. In maintenance ran:

php ./showJobs.php

0 (so no jobs in the queue). Back to Special:RecentChanges, the latest changes are not included. Back to the http://csdms.colorado.edu/wiki/Special:Version; page won't load. According to some documentation this could be a memory issue but have in my LocalSettings.php already increased this: ini_set( 'memory_limit', '100M' ); .

Included the following at the end of LocalSettings.php to capture error messages in the browser:

error_reporting( E_ALL );

ini_set( 'display_errors', 1 );

No messages show up though. Any other thoughts of what might go wrong? Help is much appreciated! --Albert Ke (talk) 03:40, 11 January 2017 (UTC)

The "4" in jobs may be that someone edited after you truncated the table and before looking at the api. You can check again with select count(*) from job;

It's very strange that Special:Version doesn't give any error even with those sentences. Maybe Apache is catching the error and not displaying it at all... Can you try looking to the error log? http://askubuntu.com/questions/14763/where-are-the-apache-and-php-log-files

I haven't figured out why the 4 jobs show up in the api. This is what I get when counting the jobs:

mysql> select count(*) from job;

+----------+

| count(*) |

+----------+

| 0 |

+----------+

1 row in set (0.00 sec)

But your last idea is great!, checked the Apache log files while looking Special:Version and found a CirrusSearch error. Checked their website and looks like ElasticSearch is not installed properly. So will work on that. That will affect the runJobs as well. Will let you know here what the outcome is.

Ok, I've inspected how the api calculates the job queue size and it's not a plain count(*) of the job queue, but a very complex logic, that can probably give false results like this...

Maybe the problem with CirrusSearch+ElasticSearch is what causes RecentChanges to not update, since after an edit the search index must be updated, and maybe it tries to update the index before updating Recent Changes. An error when updating the index can cause the Recent Changes update to not be executed at all...

I thought about the same lines regarding CirrusSearch - ElasticSearch as you did. However after solving the problem with CirrusSearch - ElasticSearch (my fold, didn't read the instructions well), RecentChanges still wasn't updating. What actually stalled the RecentChanges was the 'CheckUser' extension. Indicated in the discussion page what I did to get this extension working for 1.28.0 (https://www.mediawiki.org/wiki/Extension_talk:CheckUser).

So problem solved. Thank you Ciencia!! and MarkAHershberger for all your help in this (still having a problem with SMW Property namespace that disappeared but hope to solve that soon as well ), --Albert Ke (talk) 03:53, 12 January 2017 (UTC)

I got the same problem in version 1.26 already!

Those problems in MediaWiki 1.26 may indicate you forgot to run the update.php script after doing an upgrade.