Manual talk:Pywikibot/2019

Please use one of the communication channels listed on Manual:Pywikibot/Communication rather than using this discussion board. There is very little traffic here, so it may take a while before you get a response.

Error on installing pywikibot

Python version: v2.7.13

Using: Windows 10

I am following the installation guide on Manual:Pywikibot/Installation. But when I type the command:

'python pwb.py generate_user_files'

I get an error saying:

Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128)

What should I do to solve this error? 79brue (talk) 03:10, 25 January 2019 (UTC)

Hello, I'm not sure as I am not Windows user, but I guess:

a) this is a bug in Python v2.7.13

Then please try to upgrade your Python to 2.7.15 or 3.7.2 and try again

b) there is a problem in your environment variables or username (perhaps in diacritics or special characters?)

Then you could try to set one of PYWIKIBOT_DIR or PYWIKIBOT_DIR_PWB variables. Probably easiest way to overcome could be to use -dir:"my pywikibot dir" parameter.

If nothing helps, please send us python pwb.py version output. Dvorapa (talk) 03:34, 25 January 2019 (UTC)

Thank you for helping me, but nothing helps this error.

The output is:

Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128) 79brue (talk) 04:18, 25 January 2019 (UTC)

It definitely seems to me like the setup fault, bad username, broken/outdated Python installation or broken PATH variable. Try to add your Pywikibot installation folder to PATH environment variable and then run only pwb.py generate_user_files or pwb.py version (without "python"). If this will not help too, maybe my Windows-experienced colleagues will know more than I do Dvorapa (talk) 04:31, 25 January 2019 (UTC)

I think it's bad username. Because my username was not in ascii. I changed my username, but I don't think it will solve the error. 79brue (talk) 04:59, 25 January 2019 (UTC)

I'm not a Windows expert but the stacktrace really suggests that the problem is in your environment (or Python version, depends on the point of view). So either update to Python 3, which handles non-ascii characters better or update your environment, so that the path to your home directory and your username don't contain them (PYWIKIBOT_DIR etc. seem not to be related to this specific problem). Matěj Suchánek (talk) 15:39, 25 January 2019 (UTC)

Hello,

I get the same error when I try to generate_family_file for my website https://lagbt.wiwiland.net/ with python 3.

Traceback (most recent call last):

  File "generate_family_file.py", line 229, in <module>

   FamilyFileGenerator(*sys.argv[1:]).run()

  File "generate_family_file.py", line 55, in run

   w = self.Wiki(self.base_url)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 56, in __init__

   check_response(r)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 323, in check_response

   elif response.status == 200 and SERVER_DB_ERROR_MSG in response.text:

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 201, in text

   return self.decode(self.encoding)

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 185, in encoding

   raise self._encoding

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 167, in encoding

   self.raw.decode(self.header_encoding)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 16052: invalid continuation byte

CRITICAL: Exiting due to uncaught exception <class 'UnicodeDecodeError'>

Thanks you for your help Goultard59 (talk) 09:54, 13 March 2019 (UTC)

Your issue is different. I can reproduce it on my own PC. Would you please report it on Phabricator?

Also check if you have MediaWiki database of your wiki set properly Dvorapa (talk) 10:37, 13 March 2019 (UTC)

python3.8 pwb.py generate_user_files

Traceback (most recent call last):

  File "pwb.py", line 182, in <module>

   if not check_modules():

  File "pwb.py", line 149, in check_modules

   from setup import dependencies

  File "/home/wojtek/core/setup.py", line 35, in <module>

   from setuptools import setup

ModuleNotFoundError: No module named 'setuptools' Draco flavus (talk) 18:23, 18 August 2020 (UTC)

setuptools package is mandatory for pywikibot. Please install it with

pip install "setuptools>=20.2"

The pwb.py wrapper scripts warns about this since release Pywikibot 6.0.0  @xqt 14:35, 1 April 2021 (UTC)

Massive image upload with pywikibot

Hi all. In the last years, I've been created some tools to upload free images from (mainly) government webs in Spain (for instance, most of the images in :commons:Category:Images from Ayuntamiento de Madrid have been uploaded by me using said tools). I've developed them as Python notebooks that run either on PAWS or locally by using pywikibot. I'm thinking of turning them into a flickr2commons-like tool (that is, with a web front-end from the toolserver), but, at the moment, upload of images by means of pywikibot is a task that takes several seconds per image (I assume this is a constraint set by any kind of policy). Is there a way to speed image upload (flickr2commons massively upload images) but still using pywikibot? Any contact person or link I can use to understand how I can proceed? Thanks Discasto (talk) 08:25, 15 February 2019 (UTC)

Hello. pywikibot has builtin support for compressed files. I have no clue if you have enough permissions/capabilities after the upload but it may help in general. A look in the source of similar tools could also help, for example: Manual:Pywikibot/imagecopy.py - Also remember that bulk uploading of images is a heavy task because of the following distribution, tagging, content checking, etc. workflow each image upload triggers (sooner or later).

Also take a look at the commons wiki if you haven't already: https://commons.wikimedia.org

Regards TheAeroExperience (talk) 23:18, 18 February 2019 (UTC)

Thank you, but this is not what I'm looking for. Flickr2commons does massive uploads without any any of the burdens mentioned by you. Therefore, it's possible to do it. However, it seems as if pywikibot is not able to do it in the same way. What am I missing? Discasto (talk) 21:29, 22 February 2019 (UTC)

Sleeping for xxx seconds

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Hi, when I run "python pwb.py pagefromfile.py -force" to load pages from a file, its starts fast and a bit later says "Sleeping for 9.9 seconds". Is there a way to reduce the sleeping time safely to 5 seconds? --Sigbert (talk) 11:48, 16 June 2019 (UTC)

The edit rate is restricted on most wikis. The default setting is put_throttle = 10 which allowes 6 edits per seconds. Some sites allows an edit rate up to 10 or 12 edits per minute which gives a put_throttle of 6 or 5. There is also a command line parameter: -pt:<value> (aliases are -put_thottle, -putthrottle)  @xqt 13:24, 16 June 2019 (UTC)

"6 edits per seconds", by minute instead Framawiki (talk) 06:23, 22 June 2019 (UTC)

Sure, thanks.  @xqt 09:24, 22 June 2019 (UTC)

Yes, there is a config option in user-config.py called 'put_throttle'. However, if the server will be overloaded, it will increase the value to its needs automatically, so understand this value only as a recommendedation. Dvorapa (talk) 12:25, 16 June 2019 (UTC)

I have edited user-config.py to set noisysleep = 30.0 to get rid of most "Sleeping" messages. This means that the client will patiently wait for at least 30 seconds before complaining that the server has a delay in processing. Geertivp (talk) 23:21, 1 July 2020 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

cannot use delete.py despite being admin on a wiki

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Here's the log. I am an admin on ko.wikinews (scroll it bit down), so I have no idea why pywikibot thinks I don't hold the ultimate authority? (Well, tbh, I should never see any of these errors on wikimedia wikis because of my hat.)

pywikibot-prod revi$ git show
commit 3df33cb77efd037a561223f1599885ccce59f48b (HEAD -> master, origin/master, origin/HEAD)
pywikibot-prod revi$ python3 --version
Python 3.7.4

— Revi 10:13, 23 September 2019 (UTC)

I propos to file this bug to phab: Probably it would help to use the stable release instead of the master branch.  @xqt 10:45, 23 September 2019 (UTC)

Will file there. I am one of your weird master-tester so you don't promote bugs such as these to stable releases :P — Revi 10:49, 23 September 2019 (UTC)

Many thanks for your testings. Probably the code is weird and we need those hints for improvements.  @xqt 11:11, 23 September 2019 (UTC)

Now on https://phab.wiki/233608. — Revi 10:53, 23 September 2019 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Endless loop "Sleeping for xy seconds"

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Hello everyonef from time to time i experience an endless loop of "Sleeping for x seconds" logs that stops my applied changes in Wikidata. I have an own program based on Pywikibot and most of the time it works as expected performing the edits on WD. But from time to time i get the mentioned behavior. Interestingly, it does not depend on the data. When i restart my program i works with same same data. Maybe, someone has an idea what the problem could be or how at least i could debug it? Thank you very much in advance. Aschroet (talk) 08:20, 24 December 2019 (UTC)

I propose to open a phabricator task for this issue. We need the command line invoking the bot, the bots version (pwb.py version) and the traceback if available. Probably start your script with -v verbose option to get more Information. I think this sleeping comes from server lag.  @xqt 10:39, 24 December 2019 (UTC)

Thanks for the quick reply. Unfortunately, i use the bot programmatically; so -v is not working. However, i tried:

pywikibot.config.verbose_output = 1

pywikibot.config.debug_log = True

without success. Since i have no traceback it is really hard to debug. But assuming it is a server lag, what could i do in this case? Aschroet (talk) 06:24, 26 December 2019 (UTC)

As noted above the command line, the pywikibot version and any bot's output would help to investigate into this matter. If it related to the server lag (and I think it is) you could increase the server lag response time in your user-config.py but you shouldn't and just let the bot wait until the server has a lower response. For further informations refer https://phabricator.wikimedia.org/T170621  @xqt 10:11, 26 December 2019 (UTC)

I've got a log now:

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:10

Sleeping for 4.7 seconds, 2019-12-28 15:10:15

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:20

Sleeping for 4.7 seconds, 2019-12-28 15:10:25

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:30

Sleeping for 4.7 seconds, 2019-12-28 15:10:35

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:40

Sleeping for 4.8 seconds, 2019-12-28 15:10:45

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:50

Sleeping for 4.8 seconds, 2019-12-28 15:10:55

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:00

Sleeping for 4.8 seconds, 2019-12-28 15:11:05

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:10

Sleeping for 4.7 seconds, 2019-12-28 15:11:15

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:20

Sleeping for 4.7 seconds, 2019-12-28 15:11:25

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:30

Sleeping for 4.7 seconds, 2019-12-28 15:11:35

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:40

Aschroet (talk) 14:16, 28 December 2019 (UTC)

I am still struggling with this problem. My bot version is 3.0.2020111 and Python 3.8.

Now I have only this simple script:

import pywikibot as pywikibot1

import pywikibot as pywikibot2

commons = pywikibot1.Site('commons', 'commons')

wikidata = pywikibot2.Site('wikidata', 'wikidata')

page = pywikibot2.Page(commons, 'Category:Espenfeld')

item = pywikibot2.ItemPage.fromPage(page)

It stops in the last line with an endless loop:

Pausing due to database lag: Waiting for all: 6.35 seconds lagged.

Sleeping for 5.0 seconds, 2020-02-03 22:21:08

I am really stuck here. For me it seems that there is some kind background processes or temporary data of the bot is still existing after the previous run. But even a restart of my laptop did not help. Is there any kind of things i can do? Aschroet (talk) 21:39, 3 February 2020 (UTC)

Dear Aschcroet,

don't know if you solved it, but your question inspired me after two days of troubling. for the same issue

Did you try to set max lag to a greater value than 5 in the user-config.py file? Now it is working to me. I also setup the mintrottle to 1 and this slows the data gathering process. However, it's better than an infinite loop of sleeping. My best, Marco MAstranisci (talk) 19:57, 11 March 2021 (UTC)

Increasing maxlag is one issue forcing this problem. It works for you but prevents server overloads to be decreased and blocks other bot operators due to this issue. See phab:T243701, phab:T242081You never should increase maxlag except you have really good reasons.  @xqt 11:18, 14 March 2021 (UTC)

Nothing we can solve on pywikibot side. The problem is caused by wikidata max lag which exceeded 5 seconds and leads to retry any request after few seconds over and over again.  @xqt 22:47, 3 February 2020 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.