Topic on Project:Support desk

Jobqueue not working on upgrade to 1.31

21
Summary by MarkAHershberger

Getting errors like:

 Argument 2 passed to RefreshLinksJob::__construct() must be of the type array, string given

Solve by removing the corrupt entries from the DB:

echo "delete from mwikijob where substr(job_params, 1,1) = 's';" | sudo -u www-data php maintenance/sql.php
Sm8ps (talkcontribs)

I moved and upgraded from MW-1.23. Most things are working fine but there are hard-to-track internal errors that appear on some pages. They look like (PageName substituted):

[e4ae3e6c214d28df035bfe14] /index.php/MWiki:G2017A_J1.3_Linearit%C3%A4t/20180116P_Linearit%C3%A4t,_Geraden   TypeError from line 47 of /var/www/html/mwiki/includes/jobqueue/jobs/RefreshLinksJob.php: Argument 2 passed to RefreshLinksJob::__construct() must be of the type array, string given, called in /var/www/html/mwiki/includes/jobqueue/Job.php on line 83
  Backtrace:
  #0 /var/www/html/mwiki/includes/jobqueue/Job.php(83): RefreshLinksJob->__construct(Title, string)
  #1 /var/www/html/mwiki/includes/jobqueue/JobQueueDB.php(303): Job::factory(string, Title, string, string)
  #2 /var/www/html/mwiki/includes/jobqueue/JobQueue.php(372): JobQueueDB->doPop()
  #3 /var/www/html/mwiki/includes/jobqueue/JobQueueGroup.php(269): JobQueue->pop()
  #4 /var/www/html/mwiki/includes/jobqueue/JobRunner.php(167): JobQueueGroup->pop(integer, integer, array)
  #5 /var/www/html/mwiki/includes/MediaWiki.php(1002): JobRunner->run(array)
  #6 /var/www/html/mwiki/includes/MediaWiki.php(988): MediaWiki->triggerSyncJobs(integer, MediaWiki\Logger\LegacyLogger)
  #7 /var/www/html/mwiki/includes/MediaWiki.php(912): MediaWiki->triggerJobs()
  #8 /var/www/html/mwiki/includes/MediaWiki.php(727): MediaWiki->restInPeace(string, boolean)
  #9 /var/www/html/mwiki/includes/MediaWiki.php(750): MediaWiki->{closure}()
  #10 /var/www/html/mwiki/includes/MediaWiki.php(557): MediaWiki->doPostOutputShutdown(string)
  #11 /var/www/html/mwiki/index.php(42): MediaWiki->run()
  #12 {main}

This seems to be related to the job queue. Indeed, php maintenance/showJobs.php yields 3801 and php maintenance/runJobs.php --maxjobs 100 halts on practically the same error as above:

[a92ddaa8c4248dd5e9bff9a2] [no req]   TypeError from line 47 of /var/www/html/mwiki/includes/jobqueue/jobs/RefreshLinksJob.php: Argument 2 passed to RefreshLinksJob::__construct() must be of the type array, string given, called in /var/www/html/mwiki/includes/jobqueue/Job.php on line 83
  Backtrace:
  #0 /var/www/html/mwiki/includes/jobqueue/Job.php(83): RefreshLinksJob->__construct(Title, string)
  #1 /var/www/html/mwiki/includes/jobqueue/JobQueueDB.php(303): Job::factory(string, Title, string, string)
  #2 /var/www/html/mwiki/includes/jobqueue/JobQueue.php(372): JobQueueDB->doPop()
  #3 /var/www/html/mwiki/includes/jobqueue/JobQueueGroup.php(269): JobQueue->pop()
  #4 /var/www/html/mwiki/includes/jobqueue/JobRunner.php(167): JobQueueGroup->pop(integer, integer, array)
  #5 /var/www/html/mwiki/maintenance/runJobs.php(89): JobRunner->run(array)
  #6 /var/www/html/mwiki/maintenance/doMaintenance.php(94): RunJobs->execute()
  #7 /var/www/html/mwiki/maintenance/runJobs.php(122): require_once(string)
  #8 {main}

I have seen a similar issue in Topic:Timwwgx0uiwrqa93 as well as https://phabricator.wikimedia.org/T201541 but that did not help me any further. I am at wits' end and would be very grateful for some help. Thanks in advance!

MarkAHershberger (talkcontribs)

It looks like some incorrect data has ended up in your job queue. On my sqlite-based install, the following would show me what entries are incorrect:

echo "select job_params from job where substr(job_params, 0, 2) != 'a';" | php maintenance/sql.php

Something similar should work for you. Do you see any results? If you need me to update for MySQL, let me know.

Sm8ps (talkcontribs)

Indeed, my installation uses MySQL and I do not see how to adapt these commands.

MarkAHershberger (talkcontribs)

I just checked the documentation and the syntax for mysql is the same.

What do you get if you run the above command on the command line while in the mediawiki directory?

Sm8ps (talkcontribs)
/var/www/html/mwiki$ echo "select job_params from job where substr(job_params, 0, 2) != 'a';" | sudo -u www-data php maintenance/sql.php
 Wikimedia\Rdbms\DBQueryError: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
 Query: select job_params from job where substr(job_params, 0, 2) != 'a'
 Function: Wikimedia\Rdbms\Database::query
 Error: 1146 Table 'mwiki.job' doesn't exist (localhost)
  in /var/www/html/mwiki/includes/libs/rdbms/database/Database.php:1457
 Stack trace:
 #0 /var/www/html/mwiki/includes/libs/rdbms/database/Database.php(1427): Wikimedia\Rdbms\Database->makeQueryException('Table 'mwiki.jo...', 1146, 'select job_para...', 'Wikimedia\\Rdbms...')
 #1 /var/www/html/mwiki/includes/libs/rdbms/database/Database.php(1200): Wikimedia\Rdbms\Database->reportQueryError('Table 'mwiki.jo...', 1146, 'select job_para...', 'Wikimedia\\Rdbms...', false)
 #2 /var/www/html/mwiki/maintenance/sql.php(160): Wikimedia\Rdbms\Database->query('select job_para...')
 #3 /var/www/html/mwiki/maintenance/sql.php(151): MwSql->sqlDoQuery(Object(Wikimedia\Rdbms\DatabaseMysqli), 'select job_para...', true)
 #4 /var/www/html/mwiki/maintenance/doMaintenance.php(94): MwSql->execute()
 #5 /var/www/html/mwiki/maintenance/sql.php(205): require_once('/var/www/html/m...')
 #6 {main}

I have successfully run maintenance/update.php.

MarkAHershberger (talkcontribs)

are you using a db prefix? It looks like you might be and that could cause this error.

Sm8ps (talkcontribs)

In LocalSettings.php, I have $wgDBprefix = "mwiki";

MarkAHershberger (talkcontribs)

Could you try:

echo "select job_params from mwikijob where substr(job_params, 0, 2) != 'a';" | sudo -u www-data php maintenance/sql.php

(There is currently a bug on MW 1.33 that will prevent you from upgrading since your prefix does not end with '_', btw. It should be fixed soon.)

Sm8ps (talkcontribs)

I see the need of a prefix ending in '_' now. This was my very first Mediawiki installation dating from 1.19 and I had no experience whatsoever with databases.

The command outputs 16181 lines, the first 7468 of which are repetitions of

stdClass Object
(
    [job_params] => s:0:"";
)

The remainder contains 36 blocks like

 stdClass Object

(
    [job_params] => a:4:{s:5:"table";s:13:"templatelinks";s:9:"recursive";b:1;s:16:"rootJobSignature";s:40:"d98b41a87a8345e672bde60fc4158b54330327c4";s:16:"rootJobTimestamp";s:14:"201   7471 50730100553";}
)

1547 blocks like

stdClass Object
(
    [job_params] => a:4:{s:5:"table";s:10:"imagelinks";s:9:"recursive";b:1;s:16:"rootJobSignature";s:40:"40241d880f9c648cef0aac211fde09c23ba7bf9d";s:16:"rootJobTimestamp";s:14:"20150730100601";}
)

and 593 blocks like

stdClass Object
(
    [job_params] => a:3:{s:5:"pages";a:1:{i:10189;a:2:{i:0;i:1;i:1;s:32:"Konzepte_der_Netzwerktechnologie";}}s:16:"rootJobSignature";s:40:"2edae4fe717bb23daf117473eb17f24447eaa3d9";s:16:"rootJobTimestamp";s:14:"20190611103037";}
 )
MarkAHershberger (talkcontribs)

The rows that start with s: need to be deleted as they are causing the problem.

I was working with the sqlite syntax when I suggested that snippet and it seems to have a significant difference. Or maybe I'm misunderstanding something.

In any case, could you try the following:

echo "select count(*) from mwikijob where substr(job_params, 0, 2) = ' s';" | sudo -u www-data php maintenance/sql.php

and then

echo "select count(*) from mwikijob where substr(job_params, 0, 2) = 's:';" | sudo -u www-data php maintenance/sql.php

Which one of the above returns results?

Sm8ps (talkcontribs)

They both give the same answer but that probably is not what you were looking for?

/var/www/html/mwiki$ echo "select count(*) from mwikijob where substr(job_params, 0, 2) = ' s';" | sudo -u www-data php maintenance/sql.php
stdClass Object
(
    [count(*)] => 0
)
/var/www/html/mwiki$ echo "select count(*) from mwikijob where substr(job_params, 0, 2) = 's:';" | sudo -u www-data php maintenance/sql.php
stdClass Object
(
    [count(*)] => 0
)
Sm8ps (talkcontribs)

I just recalled that this is MariaDB but the SQL syntax should not differ from MySQL as far as I know, right?

MarkAHershberger (talkcontribs)

Right, they should be the same.

How about this:

echo "select count(*) from mwikijob where job_params = 's:0:\"\";';" | sudo -u www-data php maintenance/sql.php
Sm8ps (talkcontribs)

Right on!

/var/www/html/mwiki$ echo "select count(*) from mwikijob where job_params = 's:0:\"\";';" | sudo -u www-data php maintenance/sql.php
  stdClass Object
  (
      [count(*)] => 1867
  )

This figure also corresponds to the 7468 (=1867 x 4) lines from above.

MarkAHershberger (talkcontribs)

oh, good.

So, run

echo "delete from mwikijob where job_params = 's:0:\"\";';" | sudo -u www-data php maintenance/sql.php

and then try running the job queue again.

Sm8ps (talkcontribs)

HOORAY, it is working! showjobs.php is down to zero now.

I have been so worried about this because this error would have been totally unfixable for myself. Many, many thanks MarkAHershberger! Is there a way to physically thank you for your time and effort?

Do you know of an easy way to fix the inappropriate database prefix?

MarkAHershberger (talkcontribs)

Don't worry about the prefix as that bit is being removed and your prefix will work in future MediaWiki.

MarkAHershberger (talkcontribs)

ok, I got ahold of actual mariadb-backed mediawiki and have a working query:


echo "select count(*) from mwikijob where substr(job_params, 1,1) = 's';" | sudo -u www-data php maintenance/sql.php
Sm8ps (talkcontribs)

Please disregard the title of the above post! I had taken it from Topic:Timwwgx0uiwrqa93 as the errors looked the same. The special page Recent Changes does indeed work. I had been misguided by the comparison with MW-1.23 where the magic word numberofviews appears in the footer which is missing in MW-1.31 because it has been removed.

MarkAHershberger (talkcontribs)

I've updated the title. Please edit it if it is incorrect.

Ciencia Al Poder (talkcontribs)