Manual:Pywikibot/replace.py

From MediaWiki.org
Jump to: navigation, search
Bug blank.svg
Wikimedia Git repository has this file:
compat: replace.py
Language: English  • català • français • magyar • italiano • русский

Replace.py is part of the Python Wikipedia Robot framework.

This bot replaces text. It will retrieve information on which pages might need changes either from an XML dump or a text file, or only change a single page. To get some more information, use

python replace.py -help

If you have Windows, you may omit "python".

Overview[edit | edit source]

You may use this script basically in two ways:

  1. You write all the parameters into the command line, including the text to be replaced and the replacement. This is useful for simple tasks. For example,
    replace.py -ns:0 color colour -search:color
    will search for word "color" only in articles (ns:0), changes the lower case occurrences to "colour", asks you each time to confirm the replacement, and uses the default edit comment. Be careful, because there are cases where color must not be changed to colour (e.g. in article Cascading Style Sheets); never run such a replacement automatically unless you are at least 100% sure it is always correct! This is almost the simplest form of the command (see below at the minimal parameters). -search:color is equivalent to writing the word color into the search bar and is a fast way of gathering articles, but will not find colored and colors.
    Should either the old or the new text contain spaces, use quotation marks!
    Repeated tasks may be stored in a batch file (Windows) or shell script (Linux).
  2. You store the main parameters, including old and new text, exceptions and edit comment in a file. Several tasks (called fixes) may be stored in one file and used repeatedly. This file may be either fixes.py which is included in your Pywikipedia distribution, but is subject to change at every update, so you have to save it for yourself, or user-fixes.py that is designed for personal use, but is not included in Pywiki and may be created with generate_user_files.py. This latter one has a slightly different syntax, but has an example. This is much more efficient and flexible but needs some preparation. These two methods may be combined; however, some parameters stored in the fix will overwrite corresponding parameter given in command line.

Once you have chosen between these options, you have another decision:

  1. You make simple text replacements like the above one. This is a good way for changing words, templates, categories, section titles or names, but is not flexible. For example, the above command will replace "color", but not colors, colored and Color in upper case.
    (If you are worried only about the case, you may still type replace.py -ns:0 color colour Color Colour -search:color.)
  2. You use regular expressions (often mentioned as regexes). These seek patterns and replace them with patterns. There are some examples in fixes.py. For agglutinative and inflecting languages this is the only efficient way of spelling corrections.

Your third decision will be this:

  1. You search for pages to be modified in the live wiki. This will result in acceptable speed if you work with templates, categories or the search engine, but is usually very slow for simple iteration of pages, especially in large and medium sized wikis. So replace.py -start:! is the least recommended way of usage as it wastes your time and the resources of the server. (However, sometimes it is necessary and unavoidable if your wiki does not have dumps.)
  2. You download an XML dump of your wiki from http://dumps.wikimedia.org/ (usually xxwiki-latest-pages-articles.xml.bz2) and use it with -xml. This will speed up your bot and use your time as well as the time of your computer and the server more efficiently. This is the recommended way of searching for color, colors and colored together, because the search engine unfortunately does not handle regexes. The disadvantage of this method is that you won't find articles into which the given text has got since the composition of the last dump.
    Direct access to the dumps of your wiki is something like download:huwiki/. Change the first two letters to your language code.
    Dumps at the given link are available only for Wikimedia wikis. For other wikis, contact the maintainer of that wiki to learn if they have dumps.

And, last but not least, you face one more decision:

  1. You search for pages and modify them on the fly. This will again result in acceptable speed if you work with templates, categories or the search engine, but may be very slow if you just search for a regex in the complete namespace or wiki.
  2. You save the results to a file with -save or -savenew, while you sleep, have your lunch or write some new articles for Wikipedia. In this case you may use -always since no edit in real wiki will be made. Later in a second session you may run the fix again with -file with much less dead time.
Minimal set of parameters

At least these data should be given for the bot every time:

  1. Where and how to search for the pages to be edited?
    The corresponding parameter may be any of -start, -file, -page, -search, -xml, -cat etc.; see the Source section of the below table.
  2. The old text to be replaced and the new one to be substituted.
    This may be one or more pairs of strings or the name of a fix.
  3. It is not mandatory, but usually worth and strongly recommended for beginners to limit the work to the main namespace with -ns:0. Thus you can avoid changing the contributions of users on talk pages or correcting the title of an article on a page where the talk is just about that title. Visible part of templates (but not the code itself!) and file descriptions are also in the scope of readers. It is better not to modify talk pages, user pages and project pages (the "Wikipedia" namespace) in the first time, and it needs special care and community consensus even later. Don't be surprised of angry reactions or your bot being blocked if you omit the namespace parameter.

Files[edit | edit source]

The bot uses three files in addition to the framework:

replace.py 
the main module
fixes.py 
a few predefined "fixes"
user-fixes.py 
a file to add ones own fixes. The file is created nearly empty by generate_user_files.py

Files that may be used for input and/or output:

filename.txt 
a file with a list of articles if specified with the parameter "-file", or
a file in which the bot will save the list of articles for later use (specified with "-save"/"-savenew")
filename.xml 
a local XML dump if used with parameter "-xml"
replacelog 
the log with a name that may be specified with parameter "-log"

Parameters[edit | edit source]

Local[edit | edit source]

You can run replace.py with the following parameters (for example, python replace.py -file:articles_list.txt "errror" "error").

Source
-xml Retrieve information from a local XML dump (pages_current, see http://download.wikimedia.org). Argument can also be given as "-xml:filename".
-xmlstart Use with -xml. This will start at the given title (they are usually in order of the first edit). If you quit with Ctrl C, replace.py will write on the screen, where to continue.
-file Work on all pages given in a local text file. Will read any [[wiki link]] and use these articles. Argument can also be given as "-file:filename".
-cat Work on all pages which are in a specific category. Argument can also be given as "-cat:categoryname".
-catr Like -cat, but also recursively includes pages in subcategories, sub-subcategories etc. of the given category.
-subcats Work on all subcategories of a specific category. Argument can also be given as "-subcats:categoryname" or as "-subcats:categoryname|fromtitle".
-subcatsr Like -subcats, but also includes sub-subcategories etc. of the given category.
-transcludes Work on all pages which transclude a specific template. Argument can also be given as "-transcludes:referredtemplate", e.g. "-transcludes:stub" means transcluding stub template.
-page Only edit a specific page. Argument can also be given as "-page:pagetitle". You can give this parameter multiple times to edit multiple pages.
-ref Work on all pages that link to a certain page. Argument can also be given as "-ref:referredpagetitle".
-filelinks Works on all pages that link to a certain image. Argument can also be given as "-filelinks:ImageName".
-links Work on all pages that are linked to from a certain page. Argument can also be given as "-links:linkingpagetitle".
-start Work on all pages in the wiki, starting at a given page. Choose "-start:!" to start at the beginning. Note: You are advised to use -xml instead of this option; this is meant for cases where there is no recent XML dump.
-prefixindex Work on pages commencing with a common prefix.
-titleregex Work on pages that have titles matching the given regular expression, e.g. -titleregex:'.*foo.*'
-search Work on pages that contain the given search string e.g. -search:"Color"
Target
-save Saves the titles to a file rather than editing the article - i.e. it does a search only - clicking "[y]es" saves the filename to the file, with no change made on the wiki. This way you may collect titles to work on in automatic mode, and process them later with -file. Opens the file for append, if exists. If you insert the contents of the file into a wikipage, it will appear as a numbered list, and may be used with -links. Argument may also be given as "-save:filename". (Not supported in core version)
-savenew Just like -save, except that overwrites the existing file. Argument may also be given as "-savenew:filename".(Not supported in core version)
Replace parameters
-excepttitle:XYZ Skip pages with titles that contain XYZ. If the -regex argument is given, XYZ will be regarded as a regular expression.
-excepttext:XYZ Skip pages which contain the text XYZ. If the -regex argument is given, XYZ will be regarded as a regular expression.
-exceptinside:XYZ Skip occurrences of the to-be-replaced text which lie within XYZ. If the -regex argument is given, XYZ will be regarded as a regular expression.
-exceptinsidetag:XYZ Skip occurrences of the to-be-replaced text which lie within an XYZ tag.
Possible values of XYZ include:
comment: between '<!--' and '-->'
includeonly: between '<includeonly>' and '</includeonly>'
math: between '<math>' and '</math>'
noinclude: between '<noinclude>' and '</noinclude>'
nowiki: between '<nowiki>' and '</nowiki>' (wiki tags are ignored inside nowiki tags)
pre: between '<pre>' and '</pre>' (preformatted text)
source: between '<source ' and '</source>' (inline references)
ref: between '<ref[ >]' and '</ref>'
timeline: between '<timeline>' and '</timeline>'
startspace: lines that start with a space
table: within a table
template: between '{{' and '}}' (cascaded up to level 3, but no deeper)
hyperlink:
gallery: between '<gallery' and '</gallery>'
link: within internal wikilinks, but also interwiki, categories, and images
interwiki: within interwikis

Additional explanation is available.

-summary:XYZ Set the summary message text, bypassing the default edit summaries.
-fix:XYZ Perform one of the predefined replacements tasks, which are given in the dictionary 'fixes' defined inside the file fixes.py or user-fixes.py. The -regex argument and given replacements will be ignored if you use -fix. Currently available predefined fixes are:
  • HTML - convert HTML tags to wiki syntax, and fix XHTML.
  • syntax - try to fix bad wiki markup.
  • case-de - fix case errors in German.
  • grammar-de - fix grammar and typography in German.
-replacementfile Lines from the given file name(s) will be read as if they were added to the command line at that point. I.e. a file containing lines "a" and "b", used as python replace.py -page:X -replacementfile:file c d will replace 'a' with 'b' and 'c' with 'd'. However, using python replace.py -page:X c -replacementfile:file d will also work, and will replace 'c' with 'a' and 'b' with 'd'.
-namespace:n

abbrev. -ns:n

Number of namespace to process. The parameter can be used multiple times. It works in combination with all other parameters except for the -start parameter. (If you want to change all pages in a particular namespace, add the namespace prefix; for example, -start:User:!.)
unnamed First unnamed argument is the old text, second argument is the new text. If the -regex argument is given, the first argument will be regarded as a regular expression, and the second argument might contain expressions like \\1 or \g<name>.
Options
-always Don't prompt you for each replacement.
-recursive Recurse replacement until possible.
-nocase Use case insensitive search expressions (including regex).
-allowoverlap When occurrences of the pattern overlap, replace all of them. Warning! Don't use this option if you don't know what you're doing, because it might easily lead to infinite loops then.
-regex Make replacements using regular expressions. If this argument isn't given, the bot will make simple text replacements.
-dotall a dot (.) also matches linebreaks when using regex.
-multiline '^' and '$' will now match begin and end of each line.

Global

arg Description Default compat core
-dir:PATH Read the bot's configuration data from directory given by PATH, instead of from the default directory.
-lang:xx Set the language of the wiki you want to work on, overriding the configuration in user-config.py. xx should be the language code. user-config.py parameter: mylang
-family:xyz Set the family of the wiki you want to work on, e.g. wikipedia, wiktionary, commons, wikitravel, …. This will override the configuration in user-config.py settings. user-config.py parameter: family
-user:xyz Log in as user 'xyz' instead of the default username.
-daemonize:xyz Immediately return control to the terminal and redirect stdout and stderr to xyz (only use for bots that require no input from stdin).
-help Show a help text for the invoked script.
-log Enable the logfile. Logs will be stored in the logs subdirectory. user-config.py parameter: log ?
-log:xyz Enable the logfile, using xyz as the filename.
-nolog Disable the logfile (if it's enabled by default).
-maxlag Sets a new maxlag parameter to a number of seconds. Defer bot edits during periods of database server lag. Default is set by config.py
-debug:item
-debug
Enable the logfile and include extensive debugging data for component "item" (for all components if the second form is used).
-putthrottle:nn
-pt:nn
Set the minimum time (in seconds) the bot will wait between saving pages. user-config.py parameter: put_throttle
-verbose
-v
Make the program output more detailed messages than usual to the standard output about its current work, or progress, while it is proceeding. This may be helpful when debugging or dealing with unusual situations. not selected
-cosmeticchanges
-cc
Toggles the cosmetic_changes setting made in config.py or user_config.py to its inverse and overrules it. All other settings and restrictions are untouched. not selected
-simulate Disables writing to the server. Useful for testing and debugging of new code (if given, doesn't do any real changes, but only shows what would have been changed).
-<config_var>:n You may use all given numeric config variables as option and modify it with command line.


Examples[edit | edit source]

If you want to change templates from the old syntax, e.g. {{msg:Stub}}, to the new syntax, e.g. {{Stub}}, download an XML dump file (page table) from http://download.wikimedia.org, then use this command:

   python replace.py -xml -regex "{{msg:(.*?)}}" "{{\1}}"

You can match patterns across more than one line:

   python replace.py -regex -start:! "First line\nSecond line" ""

You can insert or append text to a page (note the replacement text has embedded new lines):

   python replace.py -regex '(?s)^(.*)$' "\1
    > ==new message==
    > blah
    > "

If you have a dump called foobar.xml and want to fix typos, e.g. Errror -> Error, use this:

   python replace.py -xml:foobar.xml "Errror" "Error"

If you have a page called 'John Doe' and want to convert HTML tags to wiki syntax, use:

   python replace.py -page:John_Doe -fix:HTML

If you run the bot without arguments you will be prompted multiple times for replacements:

   python replace.py -file:blah.txt

The script asks the user before modifying an article. It is recommended to double-check the result to be sure that the bot did not introduce errors (especially with misspelled words). It is possible to specify a set of articles with an external text file containing Wiki links :

 [[plane]]
 [[vehicle]]
 [[train]]
 [[car]]

The bot is then called using something like :

 python replace.py [global-arguments] -file:articles_list.txt "errror" "error" 

Rather than specifying regular expressions at the command line, it's preferable to add them to user-fixes.py

 python replace.py -file:articles_list.txt -fix:example2

Example: Replacing multiple paragraphs[edit | edit source]

The original text of the page Meta:Sandbox is:

This page is for any tests.

Welcome to the sandbox!

If you want to switch the statement (the second one goes before the first one), you type the following syntax:

replace.py -page:Meta:Sandbox -regex "This page is for any tests.\r\n\r\nWelcome to the sandbox!" "Welcome to the sandbox!\n\nThis page is for any tests."

To add a new line we use \n.

Example: Gathering articles[edit | edit source]

With -save one can make fake replacements, what gives the possibility to collect articles that do or do not contain a certain text. In Hungarian Wikipedia a user wanted a list of articles of animals that do not have a {{redlist}} template, and a second one of those that don't have the template but have the string "iucnredlist.org" (because they are likely to be linked to the Red List but not with a template). In the examples "Állatfajok" means "animal species":

1. Gathering articles from animal categories that contain "iucnredlist.org" but don't contain the template:

replace.py -catr:"Állatfajok" iucnredlist.org b -excepttext:"{{redlist" -save:redlist.txt

This would replace "iucnredlist.org" with a letter b, but instead of modifying pages saves their titles to redlist.txt. (Sure, we are bot owners and not vandals!)

2. Gathering articles from animal categories that contain neither "iucnredlist.org" nor the template:

replace.py -catr:"Állatfajok" e b -excepttext:iucnredlist.org -excepttext:"{{redlist" -save:redlist.txt

If both texts are on exception list, what do we search for? For example the most frequent vowel of Hungarian language, e. We don't risk too much by assuming that every article contains it. :-)

If we want to avoid listing articles with upper case {{Redlist}} templates, we need either an additional exception or regular expression.

Example: Plenty of unbolding within an article[edit | edit source]

In this article there were really lot of bolded episode titles in several tables that were to be unbolded. This is the case when you may want to use a bot for one single article and this shows the role of some interesting parameters.

What are we looking for?
  • Texts between pairs of ''',
  • which are within a table (we don't want to replace in the rest of the article!),
  • but do not contain a | character (just for safety, to make sure we are still within one cell — we might perhaps omit this),
  • paying attention to having many occurrences within a table (recursion),
  • and that the tables are wrapped to several lines (dotall),
  • and that every table opening tag should match its own closing tag and every beginning of bold text should match its own closing ''' (that's why we use ?s to make the expressions ungreedy).
What do we replace it with?

With the text parts before, between and after the boldings — these are put in parentheses to be able to refer to them with their group numbers, respectively.

The command

(It is wrapped here for readability, but you should write to one line, of course.)

replace.py -page -regex -recursive -dotall -summary:Vastagtalanítás
"(\{\|.*?)'''([^\|]*?)'''(.*?\|\})" "\1\2\3"

The bot will ask for the title since we have not given it. Using double quotation marks fits to the command line and gives the freedom to use apostrophes in the expression.

Result

Here you are. (Don't click if your computer is not strong enough!)

Advanced use of fixes: own functions[edit | edit source]

Being a wizard by means of replace.py is not a dream if you are familiar with the basics of Python programming. Textlib.py (another module of the Pywikipedia framework) has a wonderful but not widely used ability. If you write a function instead of a constant text or a regular expression to the replacement text or the exceptions, it will recognize and execute it. With a little bit of programming you may take advantage of this feature, and use replace.py at a higher level. Needless to say, using an own function gives much more flexibility than a simple regex. You may also use a function to generate the replacement expressions so as to keep them clearly arranged.

To learn how to use your own functions in fixes.py and user-fixes.py and what is this good for, see hu:Szerkesztő:Bináris/Fixes and functions HOWTO.

External links[edit | edit source]