Hello everyonef from time to time i experience an endless loop of "Sleeping for x seconds" logs that stops my applied changes in Wikidata. I have an own program based on Pywikibot and most of the time it works as expected performing the edits on WD. But from time to time i get the mentioned behavior. Interestingly, it does not depend on the data. When i restart my program i works with same same data. Maybe, someone has an idea what the problem could be or how at least i could debug it? Thank you very much in advance.
Topic on Manual talk:Pywikibot
I propose to open a phabricator task for this issue. We need the command line invoking the bot, the bots version (pwb.py version) and the traceback if available. Probably start your script with -v verbose option to get more Information. I think this sleeping comes from server lag.
Thanks for the quick reply. Unfortunately, i use the bot programmatically; so -v is not working. However, i tried:
pywikibot.config.verbose_output = 1 pywikibot.config.debug_log = True
without success. Since i have no traceback it is really hard to debug. But assuming it is a server lag, what could i do in this case?
As noted above the command line, the pywikibot version and any bot's output would help to investigate into this matter. If it related to the server lag (and I think it is) you could increase the server lag response time in your user-config.py but you shouldn't and just let the bot wait until the server has a lower response. For further informations refer https://phabricator.wikimedia.org/T170621
I've got a log now:
Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:10:10
Sleeping for 4.7 seconds, 2019-12-28 15:10:15
Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:10:20
Sleeping for 4.7 seconds, 2019-12-28 15:10:25
Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:10:30
Sleeping for 4.7 seconds, 2019-12-28 15:10:35
Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:10:40
Sleeping for 4.8 seconds, 2019-12-28 15:10:45
Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:10:50
Sleeping for 4.8 seconds, 2019-12-28 15:10:55
Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:11:00
Sleeping for 4.8 seconds, 2019-12-28 15:11:05
Pausing due to database lag: Waiting for all: 5.1 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:11:10
Sleeping for 4.7 seconds, 2019-12-28 15:11:15
Pausing due to database lag: Waiting for all: 5.1 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:11:20
Sleeping for 4.7 seconds, 2019-12-28 15:11:25
Pausing due to database lag: Waiting for all: 5.1 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:11:30
Sleeping for 4.7 seconds, 2019-12-28 15:11:35
Pausing due to database lag: Waiting for all: 5.1 seconds lagged.
Sleeping for 5.0 seconds, 2019-12-28 15:11:40
I am still struggling with this problem. My bot version is 3.0.2020111 and Python 3.8.
Now I have only this simple script:
import pywikibot as pywikibot1
import pywikibot as pywikibot2
commons = pywikibot1.Site('commons', 'commons')
wikidata = pywikibot2.Site('wikidata', 'wikidata')
page = pywikibot2.Page(commons, 'Category:Espenfeld')
item = pywikibot2.ItemPage.fromPage(page)
It stops in the last line with an endless loop:
Pausing due to database lag: Waiting for all: 6.35 seconds lagged.
Sleeping for 5.0 seconds, 2020-02-03 22:21:08
I am really stuck here. For me it seems that there is some kind background processes or temporary data of the bot is still existing after the previous run. But even a restart of my laptop did not help. Is there any kind of things i can do?
Dear Aschcroet,
don't know if you solved it, but your question inspired me after two days of troubling. for the same issue
Did you try to set max lag to a greater value than 5 in the user-config.py file? Now it is working to me. I also setup the mintrottle to 1 and this slows the data gathering process. However, it's better than an infinite loop of sleeping. My best, Marco
Increasing maxlag is one issue forcing this problem. It works for you but prevents server overloads to be decreased and blocks other bot operators due to this issue. See phab:T243701, phab:T242081You never should increase maxlag except you have really good reasons.
Nothing we can solve on pywikibot side. The problem is caused by wikidata max lag which exceeded 5 seconds and leads to retry any request after few seconds over and over again.