Manual talk:Pywikibot

Jump to navigation Jump to search

About this board

Archives 


Please use one of the communication channels listed on Manual:Pywikibot/Communication rather than using this discussion board. There is very little traffic here, so it may take a while before you get a response.

Sigbert (talkcontribs)

Hi, when I run "python pwb.py pagefromfile.py -force" to load pages from a file, its starts fast and a bit later says "Sleeping for 9.9 seconds". Is there a way to reduce the sleeping time safely to 5 seconds? --Sigbert (talk) 11:48, 16 June 2019 (UTC)

Xqt (talkcontribs)

The edit rate is restricted on most wikis. The default setting is put_throttle = 10 which allowes 6 edits per seconds. Some sites allows an edit rate up to 10 or 12 edits per minute which gives a put_throttle of 6 or 5. There is also a command line parameter: -pt:<value> (aliases are -put_thottle, -putthrottle)

Framawiki (talkcontribs)

"6 edits per seconds", by minute instead

Xqt (talkcontribs)

Sure, thanks.

Dvorapa (talkcontribs)

Yes, there is a config option in user-config.py called 'put_throttle'. However, if the server will be overloaded, it will increase the value to its needs automatically, so understand this value only as a recommendedation.

Reply to "Sleeping for xxx seconds"
79brue (talkcontribs)

Python version: v2.7.13

Using: Windows 10


I am following the installation guide on Manual:Pywikibot/Installation. But when I type the command:

'python pwb.py generate_user_files'

I get an error saying:


Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128)


What should I do to solve this error?

Dvorapa (talkcontribs)

Hello, I'm not sure as I am not Windows user, but I guess:

a) this is a bug in Python v2.7.13

Then please try to upgrade your Python to 2.7.15 or 3.7.2 and try again

b) there is a problem in your environment variables or username (perhaps in diacritics or special characters?)

Then you could try to set one of PYWIKIBOT_DIR or PYWIKIBOT_DIR_PWB variables. Probably easiest way to overcome could be to use -dir:"my pywikibot dir" parameter.

If nothing helps, please send us python pwb.py version output.

79brue (talkcontribs)

Thank you for helping me, but nothing helps this error.


The output is:


Traceback (most recent call last):

  File "pwb.py", line 177, in <module>

    import pywikibot

  File "C:\pywikibot\pywikibot\__init__.py", line 25, in <module>

    from pywikibot.bot import (

  File "C:\pywikibot\pywikibot\bot.py", line 97, in <module>

    from pywikibot import config2 as config

  File "C:\pywikibot\pywikibot\config2.py", line 392, in <module>

    base_dir = get_base_dir()

  File "C:\pywikibot\pywikibot\config2.py", line 345, in get_base_dir

    home = os.path.expanduser('~')

  File "C:\Python27\lib\ntpath.py", line 311, in expanduser

    return userhome + path[i:]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xb9 in position 9: ordinal not in range(128)

Dvorapa (talkcontribs)

It definitely seems to me like the setup fault, bad username, broken/outdated Python installation or broken PATH variable. Try to add your Pywikibot installation folder to PATH environment variable and then run only pwb.py generate_user_files or pwb.py version (without "python"). If this will not help too, maybe my Windows-experienced colleagues will know more than I do

79brue (talkcontribs)

I think it's bad username. Because my username was not in ascii. I changed my username, but I don't think it will solve the error.

Matěj Suchánek (talkcontribs)

I'm not a Windows expert but the stacktrace really suggests that the problem is in your environment (or Python version, depends on the point of view). So either update to Python 3, which handles non-ascii characters better or update your environment, so that the path to your home directory and your username don't contain them (PYWIKIBOT_DIR etc. seem not to be related to this specific problem).

Goultard59 (talkcontribs)

Hello,

I get the same error when I try to generate_family_file for my website https://lagbt.wiwiland.net/ with python 3.

Traceback (most recent call last):

  File "generate_family_file.py", line 229, in <module>

   FamilyFileGenerator(*sys.argv[1:]).run()

  File "generate_family_file.py", line 55, in run

   w = self.Wiki(self.base_url)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 56, in __init__

   check_response(r)

  File "/home/goultard/piwikibot/core/pywikibot/site_detect.py", line 323, in check_response

   elif response.status == 200 and SERVER_DB_ERROR_MSG in response.text:

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 201, in text

   return self.decode(self.encoding)

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 185, in encoding

   raise self._encoding

  File "/home/goultard/piwikibot/core/pywikibot/comms/threadedhttp.py", line 167, in encoding

   self.raw.decode(self.header_encoding)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 16052: invalid continuation byte

CRITICAL: Exiting due to uncaught exception <class 'UnicodeDecodeError'>

Thanks you for your help

Dvorapa (talkcontribs)

Your issue is different. I can reproduce it on my own PC. Would you please report it on Phabricator?

Also check if you have MediaWiki database of your wiki set properly

Reply to "Error on installing pywikibot"
Discasto (talkcontribs)

Hi all. In the last years, I've been created some tools to upload free images from (mainly) government webs in Spain (for instance, most of the images in :commons:Category:Images from Ayuntamiento de Madrid have been uploaded by me using said tools). I've developed them as Python notebooks that run either on PAWS or locally by using pywikibot. I'm thinking of turning them into a flickr2commons-like tool (that is, with a web front-end from the toolserver), but, at the moment, upload of images by means of pywikibot is a task that takes several seconds per image (I assume this is a constraint set by any kind of policy). Is there a way to speed image upload (flickr2commons massively upload images) but still using pywikibot? Any contact person or link I can use to understand how I can proceed? Thanks

TheAeroExperience (talkcontribs)

Hello. pywikibot has builtin support for compressed files. I have no clue if you have enough permissions/capabilities after the upload but it may help in general. A look in the source of similar tools could also help, for example: Manual:Pywikibot/imagecopy.py - Also remember that bulk uploading of images is a heavy task because of the following distribution, tagging, content checking, etc. workflow each image upload triggers (sooner or later).

Also take a look at the commons wiki if you haven't already: https://commons.wikimedia.org

Regards

Discasto (talkcontribs)

Thank you, but this is not what I'm looking for. Flickr2commons does massive uploads without any any of the burdens mentioned by you. Therefore, it's possible to do it. However, it seems as if pywikibot is not able to do it in the same way. What am I missing?

Reply to "Massive image upload with pywikibot"
Angrydog001 (talkcontribs)

when I run welcome.py, it returns:

Traceback (most recent call last):

  File "pwb.py", line 264, in <module>

   if not main():

  File "pwb.py", line 257, in main

   run_python_file(filename, [filename] + args, argvu, file_package)

  File "pwb.py", line 120, in run_python_file

   main_mod.__dict__)

  File ".\scripts\welcome.py", line 880, in <module>

   main()

  File ".\scripts\welcome.py", line 859, in main

   bot.run()

  File ".\scripts\welcome.py", line 611, in run

   for users in self.parseNewUserLog():

  File ".\scripts\welcome.py", line 563, in parseNewUserLog

   yield pywikibot.User(ue.page())

  File "D:\Pywikibot\core-aoa-zh\pywikibot\logentries.py", line 122, in page

   self._page = pywikibot.Page(self.site, self.data['title'])

  File "D:\Pywikibot\core-aoa-zh\pywikibot\logentries.py", line 34, in __missing__

   raise KeyError("Log entry (%s) has no '%s' key" % (self._type, key))

KeyError: "Log entry (newusers) has no 'title' key"

<class 'KeyError'>

CRITICAL: Closing network session.

Angrydog001 (talkcontribs)

What should I do?

Xqt (talkcontribs)

File this bug to phabricator

Framawiki (talkcontribs)

Hi Angrydog001, please fill a ticket on phab phab:/project/view/87/, with the command you entered to start the script, the full traceback present in your last message and the output of `python pwb.py version` command.

Miles Fides (talkcontribs)

I rather have some questions than a problem.

  • Does anyone know how the pywikibot would handle a timeout?
  • Has anybody ever experienced a timeout with the pywikibot?
  • What is the default ttl and the default reaction to a timeout?

Thanks

Zbmath authorid (talkcontribs)

I would be interested in the answer to these questions...

Reply to "Timeout handling"

How to disable "Sleeping for N seconds" on PyWikiBot

5
Summary by Dvorapa

Sleeping for N seconds should not be disabled as it makes load to servers

95.188.216.190 (talkcontribs)

How to disable "Sleeping for N seconds" on PyWikiBot

Xqt (talkcontribs)

You shouldn't. This is necessary to throttle writing to the site high frequently and it is restricted on most wiki's.

Framawiki (talkcontribs)

Why you want to disable a security ? Not a good idea I think :)

Sergezolotukhin (talkcontribs)
Xqt (talkcontribs)

maxthrottle isn't the right parameter for that.

Deleting blank lines with replace.py inside templates

6
Theklan (talkcontribs)

Hello! I have been using replace.py to delete obsolete lines inside templates, but now I have some empty lines. I can't find a way to delete them... can someone help with this? Thanks!

Matěj Suchánek (talkcontribs)

Please share more information: What is your regex, what does your input look like, what do you expect?

Theklan (talkcontribs)
Matěj Suchánek (talkcontribs)

Don't you want to replace it with a single newline \n? The problem I can see is that there were some spaces between the newlines, which you can fix with \n *\n *\n.

Theklan (talkcontribs)

Thanks for the clue! Is there any other script to delete blank lines only inside templates?

Matěj Suchánek (talkcontribs)

None that I know.

Reply to "Deleting blank lines with replace.py inside templates"
Summary by Dvorapa

Everybody can use pywikibot, but please be careful to local community restrictions

SamiIslam (talkcontribs)

Hi,

Can a normal user use the pywikibot functionality from a wiki site to create a site from a text file? Or is that something only an admin would have to do on the server hosting the wiki?

Thanks!

Sami

Xqt (talkcontribs)

There is a script "pagefromfile.py" to do this but it is may be local community restrictions to allow or discard edits in this way.

Summary by Dvorapa

The combination of movepages.py and redirect.py can effectively move pages and update redirects. There is currently no script, which could do both actions at once

Jc86035 (talkcontribs)

Does movepages.py update redirects as well to avoid creating double redirects?

This post was hidden by Jc86035 (history)
Xqt (talkcontribs)

Currently not.

Jc86035 (talkcontribs)

Are there any scripts which do so?

XXN (talkcontribs)

redirect.py

Pagegenerator in different language.

3
Summary by Dvorapa

Pagegenerator for the Czech Wikipedia is -family:wikipedia -lang:cs

Walter Klosse (talkcontribs)

How i make pagegenerator for category on wikipedia in czech language?

Framawiki (talkcontribs)

Hello, do you want something like

python pwb.py replace "1" "2" -lang:cz -category:test

?

Dvorapa (talkcontribs)

correct is -lang:cs