Manual:Pywikibot/Workshop Materials/How to host a bot on Toolforge (self-study)

This self-study guide will walk you through the process of getting access to Toolforge, setting it up for Pywikibot, and running a bot on Toolforge manually and automatically. You will learn all this using existing materials available on different wikis.

Prerequisites
Before following this self-study guide, be sure that you understand how to run Pywikibot scripts locally, on your own machine. For information on how to do that, see other small wiki toolkits workshops), Manual:Pywikibot, and How to run basic scripts (self-study).

It will also help if you have a basic understanding of Linux terminal, Bash, and SSH. The list below provides useful resources if you want to learn more about any of these subjects.


 * wikibooks:Non-Programmer's_Tutorial_for_Python_3
 * Bash_Shell_Scripting
 * Internet_Technologies/SSH

What is Toolforge?


Toolforge is a shared hosting platform supported by Wikimedia Foundation staff and volunteers. It provides users with a Linux machine they can use, for example, to run bots or host a website. Toolforge is extensively documented on wikitech:Help:Toolforge - use this portal if you have any questions or want to learn more than covered in this guide.

To use Toolforge, you must agree to its terms and conditions. For details, see wikitech:Portal:Toolforge/Quickstart#Terms_and_conditions.

Creating a developer account and setting up access to Toolforge
; ;

To use Toolforge for your bot, you need the following:


 * Wikimedia developer account (this is different from your Wikipedia or SUL account)
 * Toolforge membership
 * Tool account on Toolforge
 * SSH key you will use to log in to Toolforge

To fulfill these requirements, follow the Getting started with Toolforge - Quickstart guide.

If you need clarification on any of the steps in this process, check the list of resources below.

When creating the account, pay extra attention to the UNIX shell username. That is the username you will use when logging in to Toolforge.

Resources :


 * Generating a new SSH key
 * Instead of adding the SSH key to your developer account on Wikitech, you can do that directly in the admin console: https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/

Logging in to Toolforge


To log in to Toolforge on Linux or macOS, use the  command in your terminal, for example. On Windows, you can try the same command in the command line, PowerShell, or Git Bash (you might have SSH installed), or follow the instructions on wikitech:Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP.

It's impossible to log in to Toolforge using your developer account login and password. You can only log in to Toolforge using your UNIX shell username and SSH key. The only password you will need is the password for that key.

Resources :


 * Internet_Technologies/SSH

Setting up Pywikibot
To set up your tool account to run Pywikibot, follow the instructions on wikitech:Help:Toolforge/Pywikibot. Note that if you intend to use Toolforge to run a script that comes with Pywikibot, you should follow the instructions for installing Pywikibot from git. If you plan to run Pywikibot with a custom script you wrote yourself, you can install Pywikibot from PyPI.

Accessing your code on Toolforge
If you intend to run one of the built-in Pywikibot scripts, you can skip this section.

If you have a bot that you wrote yourself, you will want to make it available to the tool's account, for example in its  directory. The following list describes different ways to do this.


 * You can open a new file in  or   (in edit mode), and copy the contents of your script file directly into the editor in your terminal. You can do this using standard copy and paste key combinations (Ctrl+C and Ctrl+V on Windows and Linux, and Command+C and Command+V on macOS), or combinations supported by your terminal application (for example Ctrl+Shift+C and Ctrl+Shift+V or Command+Shift+C and Command+Shift+V). Some terminals also allow you to right-click and choose Copy or Paste from the context menu. After pasting your code remember to save the file in the text editor.
 * You can use the  command to copy and paste a file through SSH. See Using SCP for information on how to do this. You can also use FileZilla as described in Access to Toolforge instances with PuTTY and WinSCP - FileZilla.
 * You can commit your code into a git repository, for example on GitLab, and then run  after switching to your tool account  (using  ).

Running your bot
To run Pywikibot on Toolforge, use the jobs framework. This method ensures that your bot runs in a dedicated environment with enough resources, instead of directly using the Toolforge login machine you accessed using SSH. This is important when your bot performs many operations on more than one page. Running such a bot directly in shell could slow down the login.toolforge.org machine and make it unusable for others.

To run your bot, either immediately in the background, or automatically - according to a schedule - follow the instructions in wikitech:Help:Toolforge/Pywikibot.

When running the bot automatically using a schedule, notice that the  option uses the crontab format to specify when the bot should run. Crontab is commonly used to schedule tasks on UNIX-based operating systems. There are many helpful resources that describe it online. One such resource is https://crontab.guru, but typing "crontab run every 5 minutes" in a search engine should return plenty of useful resources.

Note that the command you run using the jobs framework doesn't need to use python or pwb directly. If your bot is more complicated, for example requires that you run preparatory commands consecutively, you can wrap all commands in a shell script and run that shell script instead.

Resources:


 * wikitech:Help:Toolforge/Jobs_framework
 * wikipedia:Cron