Manual:Pywikibot/PAWS

This document provides a quick interactive overview of Pywikibot using a 'notebook' hosted on the Wikimedia Foundation labs environment using 'PAWS' (Pywikibot: A Web Shell).

Create a Wikimedia account
To follow this walk-through, you only need an account on a Wikimedia project.

To create a Wikimedia account, see Help:Logging in. (Avoid special characters until T120066 is fixed.)

Once you have created an account, please visit https://test.wikipedia.org/ (this works around T120327).

Sign into a notebook
To start a hosted notebook, go to https://tools.wmflabs.org/paws

Click "Sign in with MediaWiki", and then click "Allow" when asked to approve "Use OAuth for Authentication".

The first time you access PAWS, you need to create a server.

Once that is completed, you will be redirected to a URL like https://tools.wmflabs.org/paws/user/ /tree

Start a terminal
To start a new interactive terminal,
 * 1) Go to your PAWS home
 * 2) click 'New' on the right hand side, and
 * 3) select 'Terminal'.

This will open a new window with the URL https://tools.wmflabs.org/paws/user/ /terminals/1, with a Linux '$' prompt.

You can bookmark this URL, and return to the terminal at any time, even after you have closed your browser or shutdown your own computer.

This terminal is not an emulator. It is a real bash shell, as part of a real Linux installation sitting on a docker container, so you can use any bash command, and use any commands available on Linux that have been installed.

To see some of the commands available, use ls.

Login to the wiki
Pywikibot needs a configuration file 'user-config.py' which PAWS automatically creates for you, with https://test.wikipedia.org/ as the default wiki.

The following command should confirm that you can log into https://test.wikipedia.org/. It uses OAuth, so there is no need to enter a password.

Create a page
To create a page, enter the following command in the terminal, replacing ' ' with your username, and pressing 'Y' when prompted to accept your changes:

You have edited the wiki. View your changes by opening https://test.wikipedia.org/wiki/User_talk: in your web browser.

You can read more about these command line scripts with the '-help' command line option.

Fetch a page
Fetching many pages is achieved with the "listpages" command.

To get the contents of the page you created in the previous section, enter the following command:

Now if you go to your PAWS files list, the saved page should be present.

A real script example
When a website used on Wikipedia changes its URL, the links on Wikipedia become outdated, and possible also dead links if the website doesnt redirect from the old URLs to the new URLs. e.g. for example, Encyclopedia Britannica (EB) has changed their links, such as moving pages from http://www.britannica.com/EBchecked/media/ to http://www.britannica.com/topic/[topic name]/images-videos/*. You can find the list of usages of the old URL on English Wikipedia at https://en.wikipedia.org/wiki/Special:LinkSearch/http://www.britannica.com/EBchecked/media. Updating all those links manually will be very time consuming. Thankfully EB has maintained redirects from their old URLs to the new URLs, so this does not need to be fixed immediately.

One way to simplify URL maintenance problems is to build templates for URLs, such as w:Template:Britannica, so the base of the URL can be modified once and the change ripples into all articles using that template. See for example the External links section of Antimatter, which includes: *

However not all EB links are using templates. e.g. Lernaean Hydra currently does not. See current revision 688137640.

In this demo, we are not going to show a solution to the above problem. Instead, here is a simpler one.

English Wikipedia currently contains links to http://britannica.com/EBchecked/ instead of http://www.britannica.com/EBchecked/ ; i.e. a 'www.' subdomain is missing in the URL.

There are currently 14 cases on English Wikipedia: https://en.wikipedia.org/wiki/Special:LinkSearch/http://britannica.com/EBchecked/

Wikipedia in other language also have this problem. e.g. there is one case on German Wikipedia: https://de.wikipedia.org/wiki/Spezial:Weblinksuche/http://britannica.com/EBchecked/

In order to fix those links, we can use Pywikibot replace.py script. In this demo we will use the '-simulate' argument to avoid writing to the wiki, as there are strict rules about automated editing of English Wikipedia.

First lets list all of the pages which link to http://britannica.com/EBchecked/.

Now we check those pages actually have the literal URL in the page; i.e. they are not using a template.

Now use replace to add the missing "www."

In PAWS, and any terminal that supports color, the diff of changes will show the added "www." in green text color, making it easier to find the proposed changes.

Inside Pywikibot
Next we will use the PAWS Python session.


 * 1) Go to your PAWS home,
 * 2) click 'New' on the right hand side, and
 * 3) select 'Python 3'.

This will open a new window.

In the text box, enter the following and in the Cell menu select 'Run'.

A new text box will appear below. Run the following to create an APISite object connected to https://test.wikipedia.org/:

Describe "site" by entering it into the new text box and selecting "Run".

It should show

Out[3]: APISite("test", "wikipedia")

Create a page object:

Check it exists by running:

It should output

VERBOSE:pywiki:Found 1 wikipedia:test processes running, including this one. Out[5]: True

Show the text on the page:

Change the page text in the object:

Save the page to the wiki:

Page Test saved INFO:pywiki:Page Test saved

The interactive Python 3 notebook allows many lines to be run together. The above could be put into one text box and Run

The log of your interactive Python session can be saved or downloaded for future reference.

Documentation
Pywikibot documentation may be found at https://doc.wikimedia.org/pywikibot/index.html. It is primarily sourced from docstrings, which can be loaded in the interactive Python 3 notebook using the Python built-in function help.

For example, to look at the arguments for the save method above, run either:

or

Editing Pywikibot scripts
The Pywikibot library and scripts are located in /srv/pwb, and are read-only.

The Pywikibot library can not be modified in PAWS.

Scripts may be modified after copying them into your PAWS home.

To run a modified "checkimages.py"
 * 1) In the terminal, enter "cp /srv/pwb/scripts/checkimages.py ~"
 * 2) Click on the file checkimages.py in your PAWS home
 * 3) Edit the code and save your modifications
 * 4) In the terminal, enter "pwb.py ~/checkimages.py -simulate ..."