Manual:Pywikibot/Workshop Materials/How to run basic scripts

This page is designed as a resource for workshop organizers who want to run a workshop about basic Pywikibot scripts. By following the instructions on this page, you will learn everything you need to know to teach others about this subject. This page features a complete script based on a past workshop, and provides many useful tips for people with little experience running such events.

Note that you do not need to follow this guide to the letter. Feel free to adapt it to your needs, and contribute back to it based on your experience.

If you want to learn more about this subject before committing to running a workshop, or need more concise materials, see User:KBach-WMF/Sandbox/SWT Workshop Materials/How to run a basic scripts (self-study).

TODO: Link to any general tips and tricks for workshop organizers if they exist somewhere.

How to use this guide?
Read the guide in its entirety to determine if the content of this workshop matches your expectations. Change whatever you feel does not match your style or is not interesting for your audience.

Next, read the relevant linked materials to make sure you know everything you need to run this workshop.

Finally, prepare for the workshop by going through the section.

Requirements
To successfully run this workshop you should know how to run scripts included with Pywikibot. This means basic familiarity with Python, Pywikibot, your terminal or command line of choice, and PAWS.

Organizer
To prepare for this workshop, follow the steps below.


 * 1) Prepare a local Pywikibot environment for the  part of the workshop.
 * 2) * Make sure that Pywikibot and its scripts are available on your local environment. The best way to do this is to download the code from the Pywikibot repository as described on Manual:Pywikibot/Installation. You will then be able to run pywikibot from the core directory by calling . The rest of this page assumes you have used this method of installing Pywikibot.
 * 3) * Run, select the wiki you want to use for your demonstrations. In this workshop we recommend working with https://test.wikimedia.org, which you can select by picking ```wikipedia```, followed by the ```test``` site code. Provide your user credentials if you prefer.
 * 4) * Pick a categories and pages on the test Wikipedia that you will use to present how to manage categories and pages. Read through the instructions on this page first and then select the best categories and pages for your presentation. Create new ones if necessary.
 * 5) Prepare the PAWS environment for the  part of the workshop.
 * 6) * Create or upload a page file that you will use when presenting how to create a new page from file.
 * 7) * Create a discussion page with a few threads you will archive using a bot.
 * 8) Make the following decisions on how you want to run the workshop.
 * 9) * Do you want the participants to follow along and perform the described activities immediately as you talk about them, or wait until a designated time to perform the activities?
 * 10) * Do you want the participants to interrupt you to ask questions, or wait for the designated time instead?
 * 11) * Do you want to record the workshop? If so, ask the participants if they consent to being recorded.
 * 12) Share your decisions during the introduction to the workshop.
 * 13) Consider if there are any prerequisites or preparations required from the participants and share them before the workshop.

Participants
Be sure to inform workshop participants about the following points they need to consider to prepare for this workshop.


 * To run Pywikibot scripts locally, participants must have Pywikibot installed. For information on how to do this, see Manual:Pywikibot, other small wiki toolkits workshops (see small wiki toolkits workshops), and other self-study guides (TODO).
 * To run Pywikibot scripts on PAWS, participants need a Wikipedia/SUL account. They can create it on any Wikipedia page.

Workshop script
Start sharing your screen, ensure the recording is enabled if you planned to record.

Introduce yourself and the topic of the workshop. Explain the organizational choices and rules you've decided upon (for example, whether the participants should ask questions immediately or later, try to follow along or not - see for details).

Present the agenda of the workshop, outlining what scripts you will run and how. Talk about the purpose of the workshop.

Working with categories
Explain that in this section you will focus on Pywikibot scripts that operate on page categories. Go through the scripts you will present.

Adding categories to pages


Explain that you can use the  script to add a category to a list of pages.

Show a page you intend to provide to the script.

Run the  script and explain what is happening.

Pywikibot asks you to provide a page with the list and then specify a category. It then adds the category to all pages linked from the page you provided, but only after you confirm you want to do this.

Show the page again, pointing out that the category has been added.

Mention that Pywikibot does not add the category directly to the page you specify, only to linked pages. Also, note that you can pass the  option if you want linked pages that do not exist to be created as well. They are skipped by default.

Sources:


 * Manual:Pywikibot/category.py
 * category script documentation on doc.wikimedia.org

Removing categories from pages


Explain that you can use the  script to remove the category from all pages with that category. Mention that this script does not allow you to specify pages to remove the category from. Its primary use is for when you want to delete a category entirely.

There is an option to use a page generator to limit the reach of this script. For example, you can choose to only remove a category from pages that only exist in another, specified category. You can read more about this in the documentation.

Demonstrate the use of this script by running  and then picking a category from the test Wikipedia to remove. The best way to do this is to remove the category you have added in the previous step.

Sources:
 * Manual:Pywikibot/category.py
 * category script documentation on doc.wikimedia.org

Creating a list of pages in a given category


Explain that you can use the  script to generate a list of pages with a specific category.

Demonstrate the use of this script by running  and then providing category name and the name of a list page to create. Show the newly created page on the wiki.

Note that you can use the  option to overwrite a page if it already exists.

Sources:
 * Manual:Pywikibot/category.py
 * category script documentation on doc.wikimedia.org

Moving pages from one category to another


Explain that you can use the  script to move pages from one category to another.

Demonstrate the use of this script by running, and showing the result of this script.

Explain that this option can be combined with other options that can be interesting to participants. You can read about them in documentation.

Sources:
 * Manual:Pywikibot/category.py
 * category script documentation on doc.wikimedia.org

Page maintenance
Scripts covered in this section provide various page maintenance utilities.

Cleaning sandbox


Explain that you can use the  script to clean up your sandbox page. By default, this script sets the sandbox to display the  template.

You can specify the  option to clean up a custom page, and use the   option to set page content to custom text instead of the default.

The documentation contains information about other interesting options, for example  and   that allow for basic scheduling of this bot.

Sources:
 * Manual:Pywikibot/clean_sandbox.py
 * clean_sandbox script documentation on doc.wikimedia.org

Checking for broken links


Explain that you can use the  script to identify broken external links on a page. Note that this script requires an additional Python package called.

Run the script using. Explain that when you run the  script for the first time, it generates a binary file that is a starting point for evaluating if links are broken. This file is created in the  directory. If you run the script again on the same page after at least a week, it will generate a list of links that are still broken, in a human-readable format.

This mechanism is in place so that you avoid removing links that are broken only temporarily.

As with other scripts, this script also provides additional options you can read about in the documentation.

Sources:
 * Manual:Pywikibot/weblinkchecker.py
 * weblinkchecker script documentation on doc.wikimedia.org

Deleting pages


Explain that you can use the  script to delete or undelete specific pages. You can specify a page directly using the  option, or use page generators to construct a list of pages to delete. For example, you can use the  option to delete pages that link to a certain page, or the   to delete pages in a category.

Run the  script to demonstrate how it works. You can follow it up with. Note that you might not have the necessary permissions to directly delete pages on test Wikipedia. In that case, the script will offer to mark the page for deletion instead.

Sources:
 * Manual:Pywikibot/delete.py
 * delete script documentation on doc.wikimedia.org

Running Pywikibot scripts using PAWS


PAWS is a Jupyter Notebook instance hosted by the Wikimedia Foundation. Jupyter Notebook is a web application that allows you to execute code and present results without needing to install anything on your computer.

PAWS has Pywikibot preinstalled. You can use it to run Pywikibot scripts in multiple ways. In this part of the workshop, you will see how to use the PAWS terminal to run built-in Pywikibot scripts. Then, you will see how to write a simple custom script in the notebook interface.

Explain that PAWS is a perfect way to run your less CPU-intensive scripts, or scripts that you don't need to schedule. It is also a great way to learn more about Pywikibot and experiment with it without having to set it up locally.

On the other hand, PAWS is not a good solution for collaborative development or longer, scheduled Pywikibot tasks.

Show the PAWS page and explain that it is a good starting point for running PAWS and learning more about it. Then demonstrate how to log in to PAWS by clicking the link on that page or accessing https://hub.paws.wmcloud.org directly.

Sources:
 * PAWS
 * PAWS/Getting_started_with_PAWS
 * PAWS/PAWS_and_Pywikibot
 * Jupyter project documentation

In terminal
Demonstrate how to launch a terminal in PAWS. To do that, scroll down the Launcher page to the Other section and select Terminal.

Note that if you closed the Launcher page, you can reopen it using the New Launcher button in the top left corner of the PAWS screen.

Explain that you can use almost exactly the same syntax to call Pywikibot from PAWS. The only difference is that you omit  in your commands.

Explain that because you logged in to PAWS using a Wikipedia account, running  in terminal will work immediately and log you in to test Wikipedia with that account. This means you can immediately start experimenting with Pywikibot.

Creating pages from a file


To create a page from a file, you need to prepare that page, place it in a file on PAWS, and then use the  script. You can use the code sample below as source of your page content.

Demonstrate how to do it directly in PAWS.

Explain the syntax of the page file. Every page must start with  and end with. The first line of the file will serve as a title and a heading of the page. The rest of the page follows regular wikitext syntax rules.

{{Codesample
 * name = page.txt
 * code = {{-start-}}

Pywikibot Workshop In MONTH YEAR

This was held on DATE

List of Participants
}}
 * Participant 1
 * Participant 2

Create the page using. Show the page on test wiki after the process is finished.

Mention that this script has a lot of additional options. You can learn more about them in the documentation.

Sources:
 * Manual:Pywikibot/pagefromfile.py
 * pagefromfile script documentation on doc.wikimedia.org

Archiving page discussion


Explain that you can use the  script to archive old discussions. Show a discussion page with threads that you intend to archive.

To automatically archive threads, you need to use a configuration template on the discussion page. One commonly used template is, documented on wikipedia:User:MiszaBot/Archive_HowTo.

To configure your page for automatic archiving, add the template at the top of the page.

Explain the meaning of template parameters (see the documentation on doc.wikimedia.org for details):
 * - the name of the page to which the archived discussions will be moved. You can generate this name dynamically using parameters described in the documentation.  is one such parameter.
 * - archiving algorithm - in this case, every thread that is at least 10 seconds old will be archived. You are using a small number here to demonstrate how the bot works. Typically, you would set this to a few weeks or months.
 * - used to dynamically generate the archive page name. You can change it manually to create a new archive page.
 * - minimum number of discussion threads to keep on the page. If the bot cannot fulfill this requirement, it will not archive any threads.
 * - minimum number of threads to archive. If the bot cannot fulfill this requirement, it will not archive any threads.

Once the template is present on the discussion page, run  and explain the script's parameters. Note that the name of the template is the last option when calling this script and has to match the template you used on the page.

Show the discussion page and point out how the number of threads decreased.

Sources:
 * Manual:Pywikibot/archivebot.py
 * archivebot script documentation on doc.wikimedia.org

Checking images


Explain that you can use the  script to automatically analyze images for problems, for example with description or license.

Run the script by calling. This simulates the check of the most recently uploaded 10 images. You can also specify a page with images that you want to check using the  option.

Sources:
 * Manual:Pywikibot/checkimages.py
 * checkimages script documentation on doc.wikimedia.org

In a notebook
Demonstrate how to create a new Jupyter notebook by opening the Launcher page and then choosing Python 3 (ipykernel) in the Notebook section.

This opens a new notebook where you can start creating your custom Pywikibot script.

Creating a simple page


Copy and paste the script below into the notebook and run it using the Run the selected cells and advance option in the toolbar.

Explain what the script does based on the comments. Open the Pywikibot Workshop page to show how the page was created on test Wikipedia.

Page generators
Mention that you can use page generators with some of the scripts you showed in this workshop. Page generators allow you to execute these scripts on a dynamically generated lists of pages. You don't have to know the names of all pages you want to modify using a bot. Instead, the generators automatically create a list of those pages based on specific criteria, for example pages in specific categories, or pages that are not watched by anyone.

To learn more about page generators, read the documentation.

Sources:
 * Manual:Pywikibot/Page Generators
 * Page generators on doc.wikimedia.org

Closing
Thank the participants and other organizers. Leave some time for final questions the participants might have. List links to additional materials on your last slide (if you are using them), and let the participants know where they can find the slides and any helpful resources.

Frequently asked questions
(questions asked by participants during previous workshops)

Resource reference
(list all links for quick access)