Manual talk:Pywikibot

Jump to navigation Jump to search

About this board

Archives 


Please use one of the communication channels listed on Manual:Pywikibot/Communication rather than using this discussion board. There is very little traffic here, so it may take a while before you get a response.

2806:106E:1E:1870:9124:3F37:C7C3:A07E (talkcontribs)

Hi. Pywikibot had been working fine. Then it started flagging errors, maybe from a python update. i updated pywikibot and python no longer flags errors. But I don't know why I can't log in. i set pywikibot all over again. any ideas? mediawiki 1.33 python 3.8.3


Traceback (most recent call last):

  File "pwb.py", line 360, in <module>

   if not main():

  File "pwb.py", line 352, in main

   run_python_file(filename,

  File "pwb.py", line 73, in run_python_file

   exec(compile(source, filename, 'exec', dont_inherit=True),

  File "./scripts/login.py", line 185, in <module>

   main()

  File "./scripts/login.py", line 167, in main

   site.login(autocreate=autocreate)

  File "/home/yutkeej/pywikibot/pywikibot/tools/__init__.py", line 1499, in wrapper

   return obj(*__args, **__kw)

  File "/home/yutkeej/pywikibot/pywikibot/site/__init__.py", line 1931, in login

   if login_manager.login(retry=True, autocreate=autocreate):

  File "/home/yutkeej/pywikibot/pywikibot/login.py", line 301, in login

   cookiedata = self.getCookie()

  File "/home/yutkeej/pywikibot/pywikibot/tools/__init__.py", line 1584, in wrapper

   return obj(*new_args, **new_kwargs)

  File "/home/yutkeej/pywikibot/pywikibot/data/api.py", line 3197, in getCookie

   and (response['messagecode'] == 'login-throttled'

KeyError: 'messagecode'

CRITICAL: Exiting due to uncaught exception <class 'KeyError'>

Dvorapa (talkcontribs)

Hi, this is an issue with CAPTCHA. Your bot account is not yet trusted by wiki administrators. You can either ask your wiki admins to assign your bot account a Bot flag, or a Confirmed account flag. The Confirmed account flag is assigned automatically when your account is 4 days old or makes more than 10 edits (en:Wikipedia:User_access_levels#Autoconfirmed_users).

Dvorapa (talkcontribs)

@Xqt Perhaps we should catch this new account captcha (and solve it) somehow too. Or we could just throw a better error message than login-throttled (which has nothing to do with captcha)

Reply to "login key error"
Sigbert (talkcontribs)

Hi, when I run "python pwb.py pagefromfile.py -force" to load pages from a file, its starts fast and a bit later says "Sleeping for 9.9 seconds". Is there a way to reduce the sleeping time safely to 5 seconds? --Sigbert (talk) 11:48, 16 June 2019 (UTC)

Xqt (talkcontribs)

The edit rate is restricted on most wikis. The default setting is put_throttle = 10 which allowes 6 edits per seconds. Some sites allows an edit rate up to 10 or 12 edits per minute which gives a put_throttle of 6 or 5. There is also a command line parameter: -pt:<value> (aliases are -put_thottle, -putthrottle)

Framawiki (talkcontribs)

"6 edits per seconds", by minute instead

Xqt (talkcontribs)

Sure, thanks.

Dvorapa (talkcontribs)

Yes, there is a config option in user-config.py called 'put_throttle'. However, if the server will be overloaded, it will increase the value to its needs automatically, so understand this value only as a recommendedation.

Geertivp (talkcontribs)

I have edited user-config.py to set noisysleep = 30.0 to get rid of most "Sleeping" messages. This means that the client will patiently wait for at least 30 seconds before complaining that the server has a delay in processing.

Reply to "Sleeping for xxx seconds"

BOT watchlist notification that triggers a PWB script

1
Revansx (talkcontribs)

The PWB bot user account on my wiki has a watchlist and receives notifications like any other user. Has anyone every written a pwb script that reads its un-read notifications and calls other scripts with properties based what pages are referred to in the un-read notifications?

Reply to "BOT watchlist notification that triggers a PWB script"

internal_api_error_Error_when_running_a_script

7
Summary by Revansx

problems were due to an experimental extension I should not have had loaded

Revansx (talkcontribs)

Hello, I have installed the latest pywikibot on a centos7 server (python3/pip3) and configured it to successfully log in to a bot account I have created on my private MW 34.x wiki (yay!), but when I try to run a basic script such as "add_text.py" I get an vague error message relating to the API. It's not a permissions error, I've solved that. Now it says, "ERROR: Detected MediaWiki API exception internal_api_error_Error". Can anyone please help me debug this? Here's the error:

[revansx@meza1 core_stable]$ python3 pwb.py login
Logged in on somewiki:en as MrBot.
[revansx@meza1 core_stable]$ python3 pwb.py add_text -cat:Some_Pages -text:'Hello World'
WARNING: API error internal_api_error_Error: [XqtN7LIvu8DlHjdVfQAmeQAAAAg] Caught exception of type Error
ERROR: Detected MediaWiki API exception internal_api_error_Error: [XqtN7LIvu8DlHjdVfQAmeQAAAAg] Caught exception of type Error
[errorclass: Error]; raising
Traceback (most recent call last):
  File "pwb.py", line 390, in <module>
    if not main():
  File "pwb.py", line 385, in main
    file_package)
  File "pwb.py", line 101, in run_python_file
    main_mod.__dict__)
  File "./scripts/add_text.py", line 360, in <module>
    main()
  File "./scripts/add_text.py", line 352, in main
    for page in generator:
  File "/home/revansx/core_stable/pywikibot/pagegenerators.py", line 1554, in CategorizedPageGenerator
    for a in category.articles(**kwargs):
  File "/home/revansx/core_stable/pywikibot/page/__init__.py", line 2983, in articles
    member_type=['page', 'file']
  File "/home/revansx/core_stable/pywikibot/tools/__init__.py", line 1790, in wrapper
    return obj(*__args, **__kw)
  File "/home/revansx/core_stable/pywikibot/site/__init__.py", line 4052, in categorymembers
    total=total, g_content=content, **cmargs)
  File "/home/revansx/core_stable/pywikibot/tools/__init__.py", line 1790, in wrapper
    return obj(*__args, **__kw)
  File "/home/revansx/core_stable/pywikibot/site/__init__.py", line 1943, in _generator
    gen = gen_class(type_arg, **req_args)
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 2921, in __init__
    QueryGenerator.__init__(self, **kwargs)
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 2456, in __init__
    self.site._paraminfo.fetch('query+' + mod for mod in self.modules)
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 395, in fetch
    self._init()
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 274, in _init
    self._fetch(self.preloaded_modules)
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 478, in _fetch
    result = request.submit()
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 2251, in submit
    self._data = super(CachedRequest, self).submit()
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 2040, in submit
    if self._internal_api_error(code, error, result):
  File "/home/revansx/core_stable/pywikibot/data/api.py", line 1910, in _internal_api_error
    raise e
pywikibot.data.api.APIMWException: internal_api_error_Error: [XqtN7LIvu8DlHjdVfQAmeQAAAAg] Caught exception of type Error
[errorclass: Error]
CRITICAL: Exiting due to uncaught exception <class 'pywikibot.data.api.APIMWException'>
Framawiki (talkcontribs)

Hello Revansx, this is definitely an internal Mediawiki error (yes, it looks like you found a bug in mediawiki :), not in pywikibot itself)

You're logging at somewiki, you can ask admin of this wiki to update its mediawiki version to try to get the bug removed. If you're sure it's using up-to-date version, please put full console log of your command (if you've stripped in above paste) and the output of python3 pwb.py version.

Revansx (talkcontribs)

I'm the admin of the wiki as well. The wiki is 1.34.1. I'm running:

  • Python 3.5.6
  • pip 20.1 from /usr/lib/python3.5/site-packages/pip (python 3.5)

And my python3 pwb.pr version response is:

$ python3 pwb.py version
Pywikibot: [https] r-pywikibot-core.git (2cf6739, g1, 2020/04/05, 16:57:37, OUTDATED)
Release version: 3.1.dev0
requests version: 2.23.0
  cacerts: /usr/lib/python3.5/site-packages/certifi/cacert.pem
    certificate test: ok
Python: 3.5.6 (default, Nov 16 2018, 15:50:58)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
PYWIKIBOT_DIR: Not set
PYWIKIBOT_DIR_PWB:
PYWIKIBOT_NO_USER_CONFIG: Not set
Config base dir: /home/revansx/core_stable
Usernames for family "somewiki":
        en: MrBot

Do you see anything specific I should update?

Matěj Suchánek (talkcontribs)

As you were told, this is beyond Pywikibot, it's an internal error of the wiki (despite the bot is triggering it). You may ask at Project:Support desk but you will need to provide the traceback of the exception thrown by the server engine.

Dvorapa (talkcontribs)

Perhaps you might try to reinstall MediaWiki as your first installation might be broken.

Revansx (talkcontribs)

I appreciate the responses. I'll do what has been recommended and reply back here with whatever the solution turned out to be. Thanks.

Revansx (talkcontribs)

problem solved. There was an permissions issue incurred by an experimental extension I have installed. I disabled the extension and my pwb worked as expected. Thanks everyone

How to upload images to commons via command line?

6
TiagoLubiana (talkcontribs)

I am trying to do this for ages, but there is a lot of old information and misdirections.


I just want to upload a file via command line. Can someone here point me to a clear reference resource where I can really learn how to do it?

sorry about the angry tone, lots of ours of failure.

~~~~

Dvorapa (talkcontribs)
TiagoLubiana (talkcontribs)

In the end it all worked by:


Installing pywikibot via pip

Adding user-config.py file to the same folder as upload.py (and the directory of the files to upload too)

Running the uploads script


I was not able to specify some things, though, like wikidata items depicted, or choose the proper license, or adding a legend. Any tips on that?

Thanks!

Dvorapa (talkcontribs)

Not everything is supported by Pywikibot currently, see T223820. You can update upload.py script or write your own script using this brief manual, this basic template and this api module if you need specific feature not yet supported by Pywikibot. You could also propose a patch to Pywikibot if you manage to fix any of these.

Hbf878 (talkcontribs)

There is a parameter that allows for adding a file description string; this will become the wiki code of the Commons page of the file that you uploaded. By the way, the best description of parameters is to be found in upload.py itself, in my opinion. It's more clear than the documentations online.

python3 pwb.py upload "path/to/some_file.png" -summary:"uploaded with pywikibot" "source code string of the file page" -ignorewarn -always -keep

So in the source code string, you can include everything that you usually find on a Commons page (in wiki syntax), like filedesc with the information template, the license-header with the license template of your choice and categories. --Hbf878 (talk) 14:56, 21 April 2020 (UTC)

TiagoLubiana (talkcontribs)

Okay, thank you for the answers!

I will take a look and try to figure a way of doing it.

Reply to "How to upload images to commons via command line?"

Endless loop "Sleeping for xy seconds"

7
Aschroet (talkcontribs)

Hello everyonef from time to time i experience an endless loop of "Sleeping for x seconds" logs that stops my applied changes in Wikidata. I have an own program based on Pywikibot and most of the time it works as expected performing the edits on WD. But from time to time i get the mentioned behavior. Interestingly, it does not depend on the data. When i restart my program i works with same same data. Maybe, someone has an idea what the problem could be or how at least i could debug it? Thank you very much in advance.

Xqt (talkcontribs)

I propose to open a phabricator task for this issue. We need the command line invoking the bot, the bots version (pwb.py version) and the traceback if available. Probably start your script with -v verbose option to get more Information. I think this sleeping comes from server lag.

Aschroet (talkcontribs)

Thanks for the quick reply. Unfortunately, i use the bot programmatically; so -v is not working. However, i tried:

pywikibot.config.verbose_output = 1
pywikibot.config.debug_log = True

without success. Since i have no traceback it is really hard to debug. But assuming it is a server lag, what could i do in this case?

Xqt (talkcontribs)

As noted above the command line, the pywikibot version and any bot's output would help to investigate into this matter. If it related to the server lag (and I think it is) you could increase the server lag response time in your user-config.py but you shouldn't and just let the bot wait until the server has a lower response. For further informations refer https://phabricator.wikimedia.org/T170621

Aschroet (talkcontribs)

I've got a log now:

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:10

Sleeping for 4.7 seconds, 2019-12-28 15:10:15

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:20

Sleeping for 4.7 seconds, 2019-12-28 15:10:25

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:30

Sleeping for 4.7 seconds, 2019-12-28 15:10:35

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:40

Sleeping for 4.8 seconds, 2019-12-28 15:10:45

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:10:50

Sleeping for 4.8 seconds, 2019-12-28 15:10:55

Pausing due to database lag: Waiting for all: 5.3166666666667 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:00

Sleeping for 4.8 seconds, 2019-12-28 15:11:05

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:10

Sleeping for 4.7 seconds, 2019-12-28 15:11:15

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:20

Sleeping for 4.7 seconds, 2019-12-28 15:11:25

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:30

Sleeping for 4.7 seconds, 2019-12-28 15:11:35

Pausing due to database lag: Waiting for all: 5.1 seconds lagged.

Sleeping for 5.0 seconds, 2019-12-28 15:11:40


Aschroet (talkcontribs)

I am still struggling with this problem. My bot version is 3.0.2020111 and Python 3.8.

Now I have only this simple script:

import pywikibot as pywikibot1
import pywikibot as pywikibot2
commons = pywikibot1.Site('commons', 'commons')
wikidata = pywikibot2.Site('wikidata', 'wikidata')
page = pywikibot2.Page(commons, 'Category:Espenfeld')
item = pywikibot2.ItemPage.fromPage(page)

It stops in the last line with an endless loop:

Pausing due to database lag: Waiting for all: 6.35 seconds lagged.
Sleeping for 5.0 seconds, 2020-02-03 22:21:08

I am really stuck here. For me it seems that there is some kind background processes or temporary data of the bot is still existing after the previous run. But even a restart of my laptop did not help. Is there any kind of things i can do?

Xqt (talkcontribs)

Nothing we can solve on pywikibot side. The problem is caused by wikidata max lag which exceeded 5 seconds and leads to retry any request after few seconds over and over again.

Reply to "Endless loop "Sleeping for xy seconds""

cannot use delete.py despite being admin on a wiki

5
-revi (talkcontribs)

Here's the log. I am an admin on ko.wikinews (scroll it bit down), so I have no idea why pywikibot thinks I don't hold the ultimate authority? (Well, tbh, I should never see any of these errors on wikimedia wikis because of my hat.)

pywikibot-prod revi$ git show
commit 3df33cb77efd037a561223f1599885ccce59f48b (HEAD -> master, origin/master, origin/HEAD)
pywikibot-prod revi$ python3 --version
Python 3.7.4
Xqt (talkcontribs)

I propos to file this bug to phab: Probably it would help to use the stable release instead of the master branch.

-revi (talkcontribs)

Will file there. I am one of your weird master-tester so you don't promote bugs such as these to stable releases :P

Xqt (talkcontribs)

Many thanks for your testings. Probably the code is weird and we need those hints for improvements.

-revi (talkcontribs)
Reply to "cannot use delete.py despite being admin on a wiki"

Error on installing pywikibot

8
79brue (talkcontribs)

Python version: v2.7.13

Using: Windows 10


I am following the installation guide on Manual:Pywikibot/Installation. But when I type the command:

'python pwb.py generate_user_files'

I get an error saying:


Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128)


What should I do to solve this error?

Dvorapa (talkcontribs)

Hello, I'm not sure as I am not Windows user, but I guess:

a) this is a bug in Python v2.7.13

Then please try to upgrade your Python to 2.7.15 or 3.7.2 and try again

b) there is a problem in your environment variables or username (perhaps in diacritics or special characters?)

Then you could try to set one of PYWIKIBOT_DIR or PYWIKIBOT_DIR_PWB variables. Probably easiest way to overcome could be to use -dir:"my pywikibot dir" parameter.

If nothing helps, please send us python pwb.py version output.

79brue (talkcontribs)

Thank you for helping me, but nothing helps this error.


The output is:


Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128)

Dvorapa (talkcontribs)

It definitely seems to me like the setup fault, bad username, broken/outdated Python installation or broken PATH variable. Try to add your Pywikibot installation folder to PATH environment variable and then run only pwb.py generate_user_files or pwb.py version (without "python"). If this will not help too, maybe my Windows-experienced colleagues will know more than I do

79brue (talkcontribs)

I think it's bad username. Because my username was not in ascii. I changed my username, but I don't think it will solve the error.

Matěj Suchánek (talkcontribs)

I'm not a Windows expert but the stacktrace really suggests that the problem is in your environment (or Python version, depends on the point of view). So either update to Python 3, which handles non-ascii characters better or update your environment, so that the path to your home directory and your username don't contain them (PYWIKIBOT_DIR etc. seem not to be related to this specific problem).

Goultard59 (talkcontribs)

Hello,

I get the same error when I try to generate_family_file for my website https://lagbt.wiwiland.net/ with python 3.

Traceback (most recent call last):

  File "generate_family_file.py", line 229, in <module>

   FamilyFileGenerator(*sys.argv[1:]).run()

  File "generate_family_file.py", line 55, in run

   w = self.Wiki(self.base_url)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 56, in __init__

   check_response(r)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 323, in check_response

   elif response.status == 200 and SERVER_DB_ERROR_MSG in response.text:

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 201, in text

   return self.decode(self.encoding)

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 185, in encoding

   raise self._encoding

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 167, in encoding

   self.raw.decode(self.header_encoding)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 16052: invalid continuation byte

CRITICAL: Exiting due to uncaught exception <class 'UnicodeDecodeError'>

Thanks you for your help

Dvorapa (talkcontribs)

Your issue is different. I can reproduce it on my own PC. Would you please report it on Phabricator?

Also check if you have MediaWiki database of your wiki set properly

Reply to "Error on installing pywikibot"

Massive image upload with pywikibot

3
Discasto (talkcontribs)

Hi all. In the last years, I've been created some tools to upload free images from (mainly) government webs in Spain (for instance, most of the images in :commons:Category:Images from Ayuntamiento de Madrid have been uploaded by me using said tools). I've developed them as Python notebooks that run either on PAWS or locally by using pywikibot. I'm thinking of turning them into a flickr2commons-like tool (that is, with a web front-end from the toolserver), but, at the moment, upload of images by means of pywikibot is a task that takes several seconds per image (I assume this is a constraint set by any kind of policy). Is there a way to speed image upload (flickr2commons massively upload images) but still using pywikibot? Any contact person or link I can use to understand how I can proceed? Thanks

TheAeroExperience (talkcontribs)

Hello. pywikibot has builtin support for compressed files. I have no clue if you have enough permissions/capabilities after the upload but it may help in general. A look in the source of similar tools could also help, for example: Manual:Pywikibot/imagecopy.py - Also remember that bulk uploading of images is a heavy task because of the following distribution, tagging, content checking, etc. workflow each image upload triggers (sooner or later).

Also take a look at the commons wiki if you haven't already: https://commons.wikimedia.org

Regards

Discasto (talkcontribs)

Thank you, but this is not what I'm looking for. Flickr2commons does massive uploads without any any of the burdens mentioned by you. Therefore, it's possible to do it. However, it seems as if pywikibot is not able to do it in the same way. What am I missing?

Reply to "Massive image upload with pywikibot"
Angrydog001 (talkcontribs)

when I run welcome.py, it returns:

Traceback (most recent call last):

  File "pwb.py", line 264, in <module>

   if not main():

  File "pwb.py", line 257, in main

   run_python_file(filename, [filename] + args, argvu, file_package)

  File "pwb.py", line 120, in run_python_file

   main_mod.__dict__)

  File ".\scripts\welcome.py", line 880, in <module>

   main()

  File ".\scripts\welcome.py", line 859, in main

   bot.run()

  File ".\scripts\welcome.py", line 611, in run

   for users in self.parseNewUserLog():

  File ".\scripts\welcome.py", line 563, in parseNewUserLog

   yield pywikibot.User(ue.page())

  File "D:\Pywikibot\core-aoa-zh\pywikibot\logentries.py", line 122, in page

   self._page = pywikibot.Page(self.site, self.data['title'])

  File "D:\Pywikibot\core-aoa-zh\pywikibot\logentries.py", line 34, in __missing__

   raise KeyError("Log entry (%s) has no '%s' key" % (self._type, key))

KeyError: "Log entry (newusers) has no 'title' key"

<class 'KeyError'>

CRITICAL: Closing network session.

Angrydog001 (talkcontribs)

What should I do?

Xqt (talkcontribs)

File this bug to phabricator

Framawiki (talkcontribs)

Hi Angrydog001, please fill a ticket on phab phab:/project/view/87/, with the command you entered to start the script, the full traceback present in your last message and the output of `python pwb.py version` command.