Manual talk:Pywikibot

Jump to navigation Jump to search

About this board

Archives 


Please use one of the communication channels listed on Manual:Pywikibot/Communication rather than using this discussion board. There is very little traffic here, so it may take a while before you get a response.

Cannot edit protected namespaces

2
Backlitt (talkcontribs)

I'm running 1.35 and use $wgNamespaceProtection to restrict edit access to several namespaces, including the mainspace. ($wgNamespaceProtection[NS_MAIN] = ['main-edit'])

When I try to make an edit with pywikibot (add_text for example), the bot logs in and reads the page properly, but I'm told that the bot lacks permissions to edit in the "Page" namespace". I've made sure that both the bot and the associated user have the ['main-edit'] = true permission.

When I remove that configuration, the bot edits just fine. It edits other namespaces that don't have the special permission structure.

Is there anything I can do to get around this, other than constantly protecting/unprotecting the namespace.

Legoktm (talkcontribs)

How is your bot authenticating? If you're using BotPasswords or OAuth, you need to make sure the "main-edit" permission is in an appropriate grant (see Manual:$wgGrantPermissions) so the bot session actually has that permission.

Reply to "Cannot edit protected namespaces"

certificate verify failed: unable to get local issuer certificate

3
Pizza0614 (talkcontribs)

HELP

I am trying to do something on the Test Wikipedia

ERROR: Traceback (most recent call last):

  File "C:\code\pywikibot\core\pywikibot\data\api.py", line 1540, in _http_request

    response = http.request(self.site, uri=uri,

  File "C:\code\pywikibot\core\pywikibot\tools\__init__.py", line 1475, in wrapper

    return obj(*__args, **__kw)

  File "C:\code\pywikibot\core\pywikibot\comms\http.py", line 251, in request

    r = fetch(baseuri, headers=headers, **kwargs)

  File "C:\code\pywikibot\core\pywikibot\tools\__init__.py", line 1475, in wrapper

    return obj(*__args, **__kw)

  File "C:\code\pywikibot\core\pywikibot\comms\http.py", line 414, in fetch

    callback(response)

  File "C:\code\pywikibot\core\pywikibot\comms\http.py", line 290, in error_handling_callback

    raise FatalServerError(str(response))

pywikibot.exceptions.FatalServerError: HTTPSConnectionPool(host='test.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

Traceback (most recent call last):

  File "C:\code\pywikibot\core\pwb.py", line 363, in <module>

    if not main():

  File "C:\code\pywikibot\core\pwb.py", line 355, in main

    run_python_file(filename,

  File "C:\code\pywikibot\core\pwb.py", line 74, in run_python_file

    exec(compile(source, filename, 'exec', dont_inherit=True),

  File ".\scripts\userscripts\code.py", line 30, in <module>

    page.text=""

  File "C:\code\pywikibot\core\pywikibot\page\__init__.py", line 572, in text

    self.botMayEdit()  # T262136, T267770

  File "C:\code\pywikibot\core\pywikibot\page\__init__.py", line 1024, in botMayEdit

    templates = self.templatesWithParams()

  File "C:\code\pywikibot\core\pywikibot\page\__init__.py", line 2076, in templatesWithParams

    titles = {t.title() for t in self.templates()}

  File "C:\code\pywikibot\core\pywikibot\page\__init__.py", line 1451, in templates

    self._templates = list(self.itertemplates(content=content))

  File "C:\code\pywikibot\core\pywikibot\data\api.py", line 2631, in __iter__

    self.data = self.request.submit()

  File "C:\code\pywikibot\core\pywikibot\data\api.py", line 1811, in submit

    response, use_get = self._http_request(use_get, uri, body, headers,

  File "C:\code\pywikibot\core\pywikibot\data\api.py", line 1540, in _http_request

    response = http.request(self.site, uri=uri,

  File "C:\code\pywikibot\core\pywikibot\tools\__init__.py", line 1475, in wrapper

    return obj(*__args, **__kw)

  File "C:\code\pywikibot\core\pywikibot\comms\http.py", line 251, in request

    r = fetch(baseuri, headers=headers, **kwargs)

  File "C:\code\pywikibot\core\pywikibot\tools\__init__.py", line 1475, in wrapper

    return obj(*__args, **__kw)

  File "C:\code\pywikibot\core\pywikibot\comms\http.py", line 414, in fetch

    callback(response)

  File "C:\code\pywikibot\core\pywikibot\comms\http.py", line 290, in error_handling_callback

    raise FatalServerError(str(response))

pywikibot.exceptions.FatalServerError: HTTPSConnectionPool(host='test.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.FatalServerError'>

Pizza0614 (talkcontribs)

Also, on the wikipedia_family.py file I tried adding this:

def ignore_certificate_error(self, code):

    return True

It didn't work.

Pizza0614 (talkcontribs)

I got it now. Here it is:

site_verify=site.verify_SSL_certificate()
site_verify=False
Speravir (talkcontribs)

I do not know whether it is just me or there is an actual issue or more, but according to commands.log every (supposed) parameter is surrounded by double quotes, or in other words: They are added before and after every space character, even when using single quotes. This reliably causes the program execution to fail at least for users of Windows commandline (cmd.exe) wherever there are spaces in the parameter values. I noticed this with the -search param where it gets even worse because double quotes are an essential part of CirrusSearch syntax, confer Help:CirrusSearch#Words, phrases, and modifiers. I had a hard time to figure out the probably right syntax for Windows, but there is still some confusing difference compared with direct search.

Let me show you an example with some search queries in Commons: With file: example image I get more than 1 million results, but file: "example image" with double quotes leads to only around 500. Additionally, using the filter intitle it will be narrowed down to 40: file: intitle:"example image".
Now with pwb listpages -family:commons -lang:commons -format:{page.loc_title} -ns:File -search:… (the program call can be shortened this way in Windows, I will from now on leave out everything but the search param):

  • With "-search:'example image'" I get an unaltered entry in commands.log, and not surprisingly this leads to a message WARNING: API warning (result): This result was truncated because it would otherwise be larger than the limit of 12,582,912 bytes. The program paused quite a period and I canceled the execution, so I did not get any search result output.
  • "-search:'""example image""'" leads to a good-looking log entry "-search:'"example image"'". But I still get the warning, on the other hand there are not toooo many lines put out. A comparison of some results suggested valid results, but the program tells me, it would have found about 600 pages (almost 100 more than with the search in Commons). Where comes the difference from?
  • Now for the (most) confusing part: Adding intitle: leads to 0 (in words: zero) results with Pywikibot! And this output comes very fast. I’d expect the input of "-search:'intitle:""example image""'", logged as "-search:'intitle:"example image"'", should get me 40 results, though.

So, long story short, depending of whether there is an issue with the Pywikibot I’d at least suggest to better document, how to use the quotes. It could be done in one place and then linked to it from all params where quotes are possible. What I think of:

  • Write “use single quotes” instead of just “quotes” (sometimes already used) and add a section especially for users of Windows command line that the whole parameter with its values has to surrounded with a pair of double quotes while doubling every quote that should be preserved. If the double quotes around every param are also added in unixoid systems then perhaps there should be an own section for the search param, as well, but this had to be tested by someone using such an OS.
  • This could be done with “expected output”, “necessary input” or so: “expected output: -param:'foo bar', necessary input: "-param:'foo bar'"”, and for the search param the first two or all three examples from above.
  • Something you do see above only implicitly: It should be pointed out to use the dedicated namespace parameter instead of the CirrusSearch equivalent (confer Help:CirrusSearch#Prefix and namespace). The search query "-search:'"example image" prefix:file:'" leads to 0 results while in Commons this query leads to the same results like above with prepended file:, check "example image" prefix:file:. For a pwb search with this prepended namespace filter ("-search:'file: example'") I get this error message (pwb 6.0.1 from 2021-03-26):
Traceback (most recent call last):
  File "C:\Programs\Netzwerk\Mediawiki-Tools\pywikibot\pwb.py", line 363, in <module>
    if not main():
  File "C:\Programs\Netzwerk\Mediawiki-Tools\pywikibot\pwb.py", line 355, in main
    run_python_file(filename,
  File "C:\Programs\Netzwerk\Mediawiki-Tools\pywikibot\pwb.py", line 74, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File ".\scripts\listpages.py", line 282, in <module>
    main()
  File ".\scripts\listpages.py", line 257, in main
    output_list += [page_fmt.output(num=i, fmt=fmt)]
  File ".\scripts\listpages.py", line 165, in output
    return fmt.format(num=num, page=self)
TypeError: unsupported format string passed to Formatter.__format__
CRITICAL: Exiting due to uncaught exception <class 'TypeError'>
Xqt (talkcontribs)
  • Quotes for command line attributes are only necessary if you have spaces in your parameter e.g. -page:"Albert Einstein". To avoid this you can use an underscore instead: -page:Albert_Einstein.
  • Quotes can be escaped, this is important for -search command which explicitly uses quotes and spaces: -search:"\"example Image\" prefix:file:" . The corresponding parameter value is '"example Image" prefix:file:' and the API search string is "example+Image"+prefix:file: as expected
  • The trunctation looks like there are too many files (> ~3000) found and the API reduces the loads the them for a single step. You may use -step parameter to avoid this; seems -step:100 looks good for it.
  • For the traceback: seems you fromatstring looks wrong, please check it. It can also be found in the log file.
Speravir (talkcontribs)

Prescript, but written last: Sorry for most of the noise …

  • “Quotes for command line attributes are only necessary if …” I know, and potential issues with this and amplier documentation are my points. The usage of _ is a good example. It is very good for page param, but not for search, confer file: ghostscript_image with file: "ghostscript image". Side note: The first version only works because of the greyspace concept in CirrusSearch. (As another lucky side note, in this case there are the expected two results with intitle:ghostscript_image.)
    Also, I cannot replace the space char with the underscore in search if I want to use more than one filter: incategory:ghostscript -intitle:ghostscript versus incategory:ghostscript_-intitle:ghostscript. Also inside of the limited search regex I cannot replace the space char (\s is not supported), and the double quotes have a special meaning there. Pywikibot’s grep param is according to docs only applicable to page titles, not the wiki source – or Do I misunderstand it? Because I just now notice there is also a titleregex parameter.
  • “Quotes can be escaped, this is important for -search command”. Yes, but the escaping with backslash usually does not work in Windows command line. Nethertheless I had been testing this before, and it did not work. But now it does! The trick is apparently really to always surround the whole param with double quotes in Windows command line. I must have missed this specific variant in my tests (I cannot exactly remember). Should be documented!
    (Interesting: "-search:'\"example image\"'" and "-search:\"example image\"" differ by the about 100 results in File namespace as told above.)
  • Truncation message: “You may use -step parameter to avoid this“ – Ah good. Alas, not documented for listpages.py (neither as script specific nor as one of the embedded filter, generator and global params) and therefore I did not know that this exists. But now that you point me to it I understand, it’s a usage of ‑<config var>:n … pause … I’m now trying …
    No, still a message for pwb listpages -family:commons -lang:commons -format:{page.loc_title} -step:100 "-search:'file: ""example image""'" (maybe I need even smaller steps), and with almost 1000 findings even more search results (cf. my first posting), because I get results from other namespaces. The variant with prefix still does not want to work (0 results). Hence again: It should strongly be suggested to use the dedicated namespace parameter.
  • For the traceback you are right: Now on a second try it did run without error. I had reused an earlier search, but introduced a spelling mistake. :-/
Reply to "Quotes, search parameter"

BOT watchlist notification that triggers a PWB script

2
Summary by Xqt

I close this Topic. Please file a Phabricator Taskif you want activly use this script.

Revansx (talkcontribs)

The PWB bot user account on my wiki has a watchlist and receives notifications like any other user. Has anyone every written a pwb script that reads its un-read notifications and calls other scripts with properties based what pages are referred to in the un-read notifications?

Xqt (talkcontribs)

There is a script watchlist.py in scripts/archive folder which can be used for this issue. Please not that this script is archived not currently maintained by developers. Please follow the steps in archive/readme.rst how to use it. You may open a phab task to recover it to the main repository and activate further development at any time.

Miles Fides (talkcontribs)

I rather have some questions than a problem.

  • Does anyone know how the pywikibot would handle a timeout?
  • Has anybody ever experienced a timeout with the pywikibot?
  • What is the default ttl and the default reaction to a timeout?

Thanks

Zbmath authorid (talkcontribs)

I would be interested in the answer to these questions...

Xqt (talkcontribs)

socket timeout values are 6.05 seconds for connection and 45 seconds for read timeout, see HTTP SETTINGS section in config2.py. If a TimeoutError occurs the scripts retries several times. All Settings can be found in SETTINGS TO AVOID SERVER OVERLOAD section config2.py.

Reply to "Timeout handling"

cannot use delete.py despite being admin on a wiki

5
Summary by Xqt

at phabricator already

-revi (talkcontribs)

Here's the log. I am an admin on ko.wikinews (scroll it bit down), so I have no idea why pywikibot thinks I don't hold the ultimate authority? (Well, tbh, I should never see any of these errors on wikimedia wikis because of my hat.)

pywikibot-prod revi$ git show
commit 3df33cb77efd037a561223f1599885ccce59f48b (HEAD -> master, origin/master, origin/HEAD)
pywikibot-prod revi$ python3 --version
Python 3.7.4
Xqt (talkcontribs)

I propos to file this bug to phab: Probably it would help to use the stable release instead of the master branch.

-revi (talkcontribs)

Will file there. I am one of your weird master-tester so you don't promote bugs such as these to stable releases :P

Xqt (talkcontribs)

Many thanks for your testings. Probably the code is weird and we need those hints for improvements.

-revi (talkcontribs)

Error on installing pywikibot

10
79brue (talkcontribs)

Python version: v2.7.13

Using: Windows 10


I am following the installation guide on Manual:Pywikibot/Installation. But when I type the command:

'python pwb.py generate_user_files'

I get an error saying:


Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128)


What should I do to solve this error?

Dvorapa (talkcontribs)

Hello, I'm not sure as I am not Windows user, but I guess:

a) this is a bug in Python v2.7.13

Then please try to upgrade your Python to 2.7.15 or 3.7.2 and try again

b) there is a problem in your environment variables or username (perhaps in diacritics or special characters?)

Then you could try to set one of PYWIKIBOT_DIR or PYWIKIBOT_DIR_PWB variables. Probably easiest way to overcome could be to use -dir:"my pywikibot dir" parameter.

If nothing helps, please send us python pwb.py version output.

79brue (talkcontribs)

Thank you for helping me, but nothing helps this error.


The output is:


Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128)

Dvorapa (talkcontribs)

It definitely seems to me like the setup fault, bad username, broken/outdated Python installation or broken PATH variable. Try to add your Pywikibot installation folder to PATH environment variable and then run only pwb.py generate_user_files or pwb.py version (without "python"). If this will not help too, maybe my Windows-experienced colleagues will know more than I do

79brue (talkcontribs)

I think it's bad username. Because my username was not in ascii. I changed my username, but I don't think it will solve the error.

Matěj Suchánek (talkcontribs)

I'm not a Windows expert but the stacktrace really suggests that the problem is in your environment (or Python version, depends on the point of view). So either update to Python 3, which handles non-ascii characters better or update your environment, so that the path to your home directory and your username don't contain them (PYWIKIBOT_DIR etc. seem not to be related to this specific problem).

Goultard59 (talkcontribs)

Hello,

I get the same error when I try to generate_family_file for my website https://lagbt.wiwiland.net/ with python 3.

Traceback (most recent call last):

  File "generate_family_file.py", line 229, in <module>

   FamilyFileGenerator(*sys.argv[1:]).run()

  File "generate_family_file.py", line 55, in run

   w = self.Wiki(self.base_url)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 56, in __init__

   check_response(r)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 323, in check_response

   elif response.status == 200 and SERVER_DB_ERROR_MSG in response.text:

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 201, in text

   return self.decode(self.encoding)

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 185, in encoding

   raise self._encoding

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 167, in encoding

   self.raw.decode(self.header_encoding)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 16052: invalid continuation byte

CRITICAL: Exiting due to uncaught exception <class 'UnicodeDecodeError'>

Thanks you for your help

Dvorapa (talkcontribs)

Your issue is different. I can reproduce it on my own PC. Would you please report it on Phabricator?

Also check if you have MediaWiki database of your wiki set properly

Draco flavus (talkcontribs)

python3.8 pwb.py generate_user_files

Traceback (most recent call last):

  File "pwb.py", line 182, in <module>

   if not check_modules():

  File "pwb.py", line 149, in check_modules

   from setup import dependencies

  File "/home/wojtek/core/setup.py", line 35, in <module>

   from setuptools import setup

ModuleNotFoundError: No module named 'setuptools'


~~~~

Xqt (talkcontribs)

setuptools package is mandatory for pywikibot. Please install it with


pip install "setuptools>=20.2"


The pwb.py wrapper scripts warns about this since release Pywikibot 6.0.0

Reply to "Error on installing pywikibot"

Sleeping for xxx seconds

6
Summary by Xqt

Nothing left to do here

Sigbert (talkcontribs)

Hi, when I run "python pwb.py pagefromfile.py -force" to load pages from a file, its starts fast and a bit later says "Sleeping for 9.9 seconds". Is there a way to reduce the sleeping time safely to 5 seconds? --Sigbert (talk) 11:48, 16 June 2019 (UTC)

Xqt (talkcontribs)

The edit rate is restricted on most wikis. The default setting is put_throttle = 10 which allowes 6 edits per seconds. Some sites allows an edit rate up to 10 or 12 edits per minute which gives a put_throttle of 6 or 5. There is also a command line parameter: -pt:<value> (aliases are -put_thottle, -putthrottle)

Framawiki (talkcontribs)

"6 edits per seconds", by minute instead

Xqt (talkcontribs)

Sure, thanks.

Dvorapa (talkcontribs)

Yes, there is a config option in user-config.py called 'put_throttle'. However, if the server will be overloaded, it will increase the value to its needs automatically, so understand this value only as a recommendedation.

Geertivp (talkcontribs)

I have edited user-config.py to set noisysleep = 30.0 to get rid of most "Sleeping" messages. This means that the client will patiently wait for at least 30 seconds before complaining that the server has a delay in processing.

Summary by Xqt

Moved to phab

2806:106E:1E:1870:9124:3F37:C7C3:A07E (talkcontribs)

Hi. Pywikibot had been working fine. Then it started flagging errors, maybe from a python update. i updated pywikibot and python no longer flags errors. But I don't know why I can't log in. i set pywikibot all over again. any ideas? mediawiki 1.33 python 3.8.3


Traceback (most recent call last):

  File "pwb.py", line 360, in <module>

   if not main():

  File "pwb.py", line 352, in main

   run_python_file(filename,

  File "pwb.py", line 73, in run_python_file

   exec(compile(source, filename, 'exec', dont_inherit=True),

  File "./scripts/login.py", line 185, in <module>

   main()

  File "./scripts/login.py", line 167, in main

   site.login(autocreate=autocreate)

  File "/home/yutkeej/pywikibot/pywikibot/tools/__init__.py", line 1499, in wrapper

   return obj(*__args, **__kw)

  File "/home/yutkeej/pywikibot/pywikibot/site/__init__.py", line 1931, in login

   if login_manager.login(retry=True, autocreate=autocreate):

  File "/home/yutkeej/pywikibot/pywikibot/login.py", line 301, in login

   cookiedata = self.getCookie()

  File "/home/yutkeej/pywikibot/pywikibot/tools/__init__.py", line 1584, in wrapper

   return obj(*new_args, **new_kwargs)

  File "/home/yutkeej/pywikibot/pywikibot/data/api.py", line 3197, in getCookie

   and (response['messagecode'] == 'login-throttled'

KeyError: 'messagecode'

CRITICAL: Exiting due to uncaught exception <class 'KeyError'>

Dvorapa (talkcontribs)

Hi, this is an issue with CAPTCHA. Your bot account is not yet trusted by wiki administrators. You can either ask your wiki admins to assign your bot account a Bot flag, or a Confirmed account flag. The Confirmed account flag is assigned automatically when your account is 4 days old or makes more than 10 edits (en:Wikipedia:User_access_levels#Autoconfirmed_users).

Dvorapa (talkcontribs)

@Xqt Perhaps we should catch this new account captcha (and solve it) somehow too. Or we could just throw a better error message than login-throttled (which has nothing to do with captcha)

Xqt (talkcontribs)

Endless loop "Sleeping for xy seconds"

9
Summary by Xqt

Nothing can be done here

Aschroet (talkcontribs)

Hello everyonef from time to time i experience an endless loop of "Sleeping for x seconds" logs that stops my applied changes in Wikidata. I have an own program based on Pywikibot and most of the time it works as expected performing the edits on WD. But from time to time i get the mentioned behavior. Interestingly, it does not depend on the data. When i restart my program i works with same same data. Maybe, someone has an idea what the problem could be or how at least i could debug it? Thank you very much in advance.

Xqt (talkcontribs)

I propose to open a phabricator task for this issue. We need the command line invoking the bot, the bots version (pwb.py version) and the traceback if available. Probably start your script with -v verbose option to get more Information. I think this sleeping comes from server lag.

Aschroet (talkcontribs)

Thanks for the quick reply. Unfortunately, i use the bot programmatically; so -v is not working. However, i tried:

pywikibot.config.verbose_output = 1
pywikibot.config.debug_log = True

without success. Since i have no traceback it is really hard to debug. But assuming it is a server lag, what could i do in this case?

Xqt (talkcontribs)

As noted above the command line, the pywikibot version and any bot's output would help to investigate into this matter. If it related to the server lag (and I think it is) you could increase the server lag response time in your user-config.py but you shouldn't and just let the bot wait until the server has a lower response. For further informations refer https://phabricator.wikimedia.org/T170621

Aschroet (talkcontribs)

I've got a log now:

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:10

Sleeping for 4.7 seconds, 2019-12-28 15:10:15

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:20

Sleeping for 4.7 seconds, 2019-12-28 15:10:25

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:30

Sleeping for 4.7 seconds, 2019-12-28 15:10:35

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:40

Sleeping for 4.8 seconds, 2019-12-28 15:10:45

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:50

Sleeping for 4.8 seconds, 2019-12-28 15:10:55

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:00

Sleeping for 4.8 seconds, 2019-12-28 15:11:05

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:10

Sleeping for 4.7 seconds, 2019-12-28 15:11:15

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:20

Sleeping for 4.7 seconds, 2019-12-28 15:11:25

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:30

Sleeping for 4.7 seconds, 2019-12-28 15:11:35

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:40


Aschroet (talkcontribs)

I am still struggling with this problem. My bot version is 3.0.2020111 and Python 3.8.

Now I have only this simple script:

import pywikibot as pywikibot1
import pywikibot as pywikibot2
commons = pywikibot1.Site('commons', 'commons')
wikidata = pywikibot2.Site('wikidata', 'wikidata')
page = pywikibot2.Page(commons, 'Category:Espenfeld')
item = pywikibot2.ItemPage.fromPage(page)

It stops in the last line with an endless loop:

Pausing due to database lag: Waiting for all: 6.35 seconds lagged.
Sleeping for 5.0 seconds, 2020-02-03 22:21:08

I am really stuck here. For me it seems that there is some kind background processes or temporary data of the bot is still existing after the previous run. But even a restart of my laptop did not help. Is there any kind of things i can do?

MAstranisci (talkcontribs)

Dear Aschcroet,

don't know if you solved it, but your question inspired me after two days of troubling. for the same issue

Did you try to set max lag to a greater value than 5 in the user-config.py file? Now it is working to me. I also setup the mintrottle to 1 and this slows the data gathering process. However, it's better than an infinite loop of sleeping. My best, Marco

Xqt (talkcontribs)

Increasing maxlag is one issue forcing this problem. It works for you but prevents server overloads to be decreased and blocks other bot operators due to this issue. See phab:T243701, phab:T242081You never should increase maxlag except you have really good reasons.

Xqt (talkcontribs)

Nothing we can solve on pywikibot side. The problem is caused by wikidata max lag which exceeded 5 seconds and leads to retry any request after few seconds over and over again.