Topic on Project:Support desk

I run a wiki on an Apache webserver with Mysql in a Linux-VM . This VM crashed yesterday without reason and now this is happening:

*Apache is running

*MySQL opens 200+ threads with Mysqld --daemonize myslq.pid

**It tells me, that there are 14 open files

**mysqladmin says, that 140+ localhost/wiki process are sleeping (this number is constantly growing, which causes a "to many connection" error after a while)

*My VM is reaching its RAM and CPU limitation due to MySQL and freezes

*If I try to connect to my wiki via internet it either gives me a connection time out or "Wikimedia\Rdbms\DBReadOnlyError" (but the wiki wasn't in read only mode when the vm crashed)

Since I installed MediaWiki two years ago I kept it up-to-date, made two successful upgrades to the recent versions and never changed the apache/mysql default settings (with exclusion of user/password) and it all worked fine. Until yesterday.

Further testing/logging within the wiki isn't possible due to the nature of the problem. I can't provide any log files.

I have to correct myself: After many reloads I finally got an error log.

And again: I have no idea why it says "Read-only". I checked the LocalSettings and it's not read only.

MediaWiki internal error.

Original exception: [3b46c182321d3ea4b8b9902a] /wiki/Mainpage Wikimedia\Rdbms\DBReadOnlyError from line 1121 of /var/lib/mediawiki/includes/libs/rdbms/database/Database.php: Database is read-only

Backtrace:

#0 /var/lib/mediawiki/includes/libs/rdbms/database/Database.php(2010): Wikimedia\Rdbms\Database->query(string, string)

#1 /var/lib/mediawiki/includes/objectcache/SqlBagOStuff.php(414): Wikimedia\Rdbms\Database->update(string, array, array, string)

#2 /var/lib/mediawiki/includes/libs/objectcache/BagOStuff.php(316): SqlBagOStuff->cas(array, string, array, integer)

#3 /var/lib/mediawiki/includes/objectcache/SqlBagOStuff.php(488): BagOStuff->mergeViaCas(string, Closure, integer, integer)

#4 /var/lib/mediawiki/includes/libs/objectcache/WANObjectCache.php(549): SqlBagOStuff->merge(string, Closure, integer, integer)

#5 /var/lib/mediawiki/includes/cache/MessageCache.php(732): WANObjectCache->set(string, array, integer)

#6 /var/lib/mediawiki/includes/cache/MessageCache.php(671): MessageCache->setValidationHash(string, array)

#7 /var/lib/mediawiki/includes/cache/MessageCache.php(420): MessageCache->saveToCaches(array, string, string)

#8 /var/lib/mediawiki/includes/cache/MessageCache.php(342): MessageCache->loadFromDBWithLock(string, array, NULL)

#9 /var/lib/mediawiki/includes/cache/MessageCache.php(971): MessageCache->load(string)

#10 /var/lib/mediawiki/includes/cache/MessageCache.php(928): MessageCache->getMsgFromNamespace(string, string)

#11 /var/lib/mediawiki/includes/cache/MessageCache.php(868): MessageCache->getMessageForLang(Language, string, boolean, array)

#12 /var/lib/mediawiki/includes/cache/MessageCache.php(808): MessageCache->getMessageFromFallbackChain(Language, string, boolean)

#13 /var/lib/mediawiki/includes/Message.php(1290): MessageCache->get(string, boolean, Language)

#14 /var/lib/mediawiki/includes/Message.php(1006): Message->fetchMessage()

#15 /var/lib/mediawiki/includes/specialpage/SpecialPage.php(840): Message->isDisabled()

#16 /var/lib/mediawiki/includes/specials/SpecialRecentchanges.php(164): SpecialPage->addHelpLink(string, boolean)

#17 /var/lib/mediawiki/includes/specialpage/SpecialPage.php(565): SpecialRecentChanges->execute(NULL)

#18 /var/lib/mediawiki/includes/specialpage/SpecialPageFactory.php(568): SpecialPage->run(NULL)

#19 /var/lib/mediawiki/includes/MediaWiki.php(288): SpecialPageFactory::executePath(Title, RequestContext)

#20 /var/lib/mediawiki/includes/MediaWiki.php(861): MediaWiki->performRequest()

#21 /var/lib/mediawiki/includes/MediaWiki.php(524): MediaWiki->main()

#22 /var/lib/mediawiki/index.php(42): MediaWiki->run()

#23 {main}

Exception caught inside exception handler: [3b46c182321d3ea4b8b9902a] /wiki/Mainpage Wikimedia\Rdbms\DBReadOnlyError from line 1121 of /var/lib/mediawiki/includes/libs/rdbms/database/Database.php: Database is read-only

Backtrace:

#0 /var/lib/mediawiki/includes/libs/rdbms/database/Database.php(2010): Wikimedia\Rdbms\Database->query(string, string)

#1 /var/lib/mediawiki/includes/objectcache/SqlBagOStuff.php(414): Wikimedia\Rdbms\Database->update(string, array, array, string)

#2 /var/lib/mediawiki/includes/libs/objectcache/BagOStuff.php(316): SqlBagOStuff->cas(array, string, array, integer)

#3 /var/lib/mediawiki/includes/objectcache/SqlBagOStuff.php(488): BagOStuff->mergeViaCas(string, Closure, integer, integer)

#4 /var/lib/mediawiki/includes/libs/objectcache/WANObjectCache.php(549): SqlBagOStuff->merge(string, Closure, integer, integer)

#5 /var/lib/mediawiki/includes/cache/MessageCache.php(732): WANObjectCache->set(string, array, integer)

#6 /var/lib/mediawiki/includes/cache/MessageCache.php(671): MessageCache->setValidationHash(string, array)

#7 /var/lib/mediawiki/includes/cache/MessageCache.php(420): MessageCache->saveToCaches(array, string, string)

#8 /var/lib/mediawiki/includes/cache/MessageCache.php(342): MessageCache->loadFromDBWithLock(string, array, NULL)

#9 /var/lib/mediawiki/includes/cache/MessageCache.php(971): MessageCache->load(string)

#10 /var/lib/mediawiki/includes/cache/MessageCache.php(928): MessageCache->getMsgFromNamespace(string, string)

#11 /var/lib/mediawiki/includes/cache/MessageCache.php(868): MessageCache->getMessageForLang(Language, string, boolean, array)

#12 /var/lib/mediawiki/includes/cache/MessageCache.php(808): MessageCache->getMessageFromFallbackChain(Language, string, boolean)

#13 /var/lib/mediawiki/includes/Message.php(1290): MessageCache->get(string, boolean, Language)

#14 /var/lib/mediawiki/includes/Message.php(982): Message->fetchMessage()

#15 /var/lib/mediawiki/includes/skins/Skin.php(1348): Message->exists()

#16 /var/lib/mediawiki/includes/skins/Skin.php(1305): Skin->addToSidebarPlain(array, string)

#17 /var/lib/mediawiki/includes/skins/Skin.php(1261): Skin->addToSidebar(array, string)

#18 /var/lib/mediawiki/includes/skins/Skin.php(1287): Skin->{closure}()

#19 /var/lib/mediawiki/includes/skins/SkinTemplate.php(481): Skin->buildSidebar()

#20 /var/lib/mediawiki/includes/skins/SkinTemplate.php(249): SkinTemplate->prepareQuickTemplate()

#21 /var/lib/mediawiki/includes/OutputPage.php(2388): SkinTemplate->outputPage()

#22 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(137): OutputPage->output()

#23 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(54): MWExceptionRenderer::reportHTML(Wikimedia\Rdbms\DBReadOnlyError)

#24 /var/lib/mediawiki/includes/exception/MWExceptionHandler.php(75): MWExceptionRenderer::output(Wikimedia\Rdbms\DBReadOnlyError, integer)

#25 /var/lib/mediawiki/includes/exception/MWExceptionHandler.php(149): MWExceptionHandler::report(Wikimedia\Rdbms\DBReadOnlyError)

#26 /var/lib/mediawiki/includes/MediaWiki.php(551): MWExceptionHandler::handleException(Wikimedia\Rdbms\DBReadOnlyError)

#27 /var/lib/mediawiki/index.php(42): MediaWiki->run()

#28 {main}

Update: After using the "SET GLOBAL read_only = OFF;" command in mysql, I don't get any time outs anymore but blank pages without any information.

Update: This is the last entry of the Apache errorlog.

[Fri Apr 03 19:42:38.978709 2020] [php7:error] [pid 22088] [client 46.229.168.151:31960] PHP Fatal error: Uncaught Error: Call to a member function getCode() on null in /var/lib/mediawiki/includes/user/User.php:1737\nStack trace:\n#0 /var/lib/mediawiki/includes/user/User.php(5417): User::getDefaultOptions()\n#1 /var/lib/mediawiki/includes/user/User.php(3101): User->loadOptions()\n#2 /var/lib/mediawiki/includes/context/RequestContext.php(336): User->getOption('language')\n#3 /var/lib/mediawiki/includes/Message.php(380): RequestContext->getLanguage()\n#4 /var/lib/mediawiki/includes/Message.php(1290): Message->getLanguage()\n#5 /var/lib/mediawiki/includes/Message.php(842): Message->fetchMessage()\n#6 /var/lib/mediawiki/includes/Message.php(934): Message->toString('text')\n#7 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(201): Message->text()\n#8 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(301): MWExceptionRenderer::msg('dberr-problems', 'Sorry! This sit...')\n#9 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(52): MWExceptionRenderer::reportOutageHTML(Object(Wikimedia\Rdbms\DBCon in /var/lib/mediawiki/includes/user/User.php on line 1737

Do you have a database backup? If you do, you should drop the database and restore from the backup.

If you don't, then your best bet is to try to use rebuild or repair the DB.

yeah, this sounds like a mysql error not a mediawiki error

The database checked ok. I narrowed down the error to PHP, because in some brief moments I somehow get the Mainpage with my content; but most of the time I get a blank page. I don't get any php error logs thou. Only blank pages even with error reporting on.

Here is a snippet from the Apache error log. It basically contains only this message since the VM crash:

[Wed Apr 01 06:18:42.710098 2020] [php7:error] [pid 27133] [client 114.119.161.74:12918] PHP Fatal error:  Uncaught Error: Call to a member function getCode() on null in /var/lib/mediawiki/includes/user/User.php:1737\nStack trace:\n#0 /var/lib/mediawiki/includes/user/User.php(5417): User::getDefaultOptions()\n#1 /var/lib/mediawiki/includes/user/User.php(3101): User->loadOptions()\n#2 /var/lib/mediawiki/includes/context/RequestContext.php(336): User->getOption('language')\n#3 /var/lib/mediawiki/includes/Message.php(380): RequestContext->getLanguage()\n#4 /var/lib/mediawiki/includes/Message.php(1290): Message->getLanguage()\n#5 /var/lib/mediawiki/includes/Message.php(842): Message->fetchMessage()\n#6 /var/lib/mediawiki/includes/Message.php(934): Message->toString('text')\n#7 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(201): Message->text()\n#8 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(301): MWExceptionRenderer::msg('dberr-problems', 'Sorry! This sit...')\n#9 /var/lib/mediawiki/includes/exception/MWExceptionRenderer.php(52): MWExceptionRenderer::reportOutageHTML(Object(Wikimedia\\Rdbms\\DBCon in /var/lib/mediawiki/includes/user/User.php on line 1737

Maybe one of the MediaWiki-php's got fried in the crash event?

thats just an error in displaying the page that reports the db is broken.

This is definitely primarily your database being broken plus an additional issue with the code that turns that into a friendly error message being also broken. However the primary issue is that your database is broken

After much tinkering and reconsidering, I'm going to rebuild the Database via the last backup I luckely took half a day before the crash. But if there's a bigger problem within the database, this only will create an endless loop of getting the wiki back online and losing it half a day later.

Welp, it kind of worked. Now it takes a considerable time (about 10 odd mins) to connect to my mainpage. And I get aborted connection errors from php/apache in the mysql log.