Manual:Pywikibot/Installation/Toolforge

Running your bot on Toolforge
Start by reading about the 1>wikitech:Portal:Toolforge|Toolforge environment.

Next, see the Cloud Services Getting Started guide for instructions on creating the necessary accounts and joining the Toolforge project.

Once you have been granted access to the Toolforge project, you can create a tool account to hold your bot's source code and configuration.

If you used the toolserver in the past and know how everything used to work there, see migrating from toolserver for more info on moving to the Toolforge environment.

Now you are ready to start. Login to Toolforge bastion server:

$ ssh USERNAME@tools-login.wmflabs.org

switch to the tool account with

maintainer@tools-login:~$ become toolname tools.toolname@tools-login:~$

Now install/clone the pywikibot code to your tool account as described below.

Install the bot code
Similar to the instructions given in [http://lists.wikimedia.org/pipermail/pywikipedia-l/2013-August/008168.html this mail] do:

$ git clone --recursive https://gerrit.wikimedia.org/r/pywikibot/core.git pywikibot-core $ cd pywikibot-core

Now you have to setup pywikibot. Choose any one of the following processes to configure your system:
 * Execute
 * Run your favorite bot script (e.g. ) since you are doing this in a fresh clone, it will trigger a bunch of questions on how you want to configure your local copy, answer them carefully in order to proceed.
 * If you already have a config file(s) from a previous version, you can copy those existing files into the right places (e.g. pywikibot-compat/).

Further things you might have to do (depending on what bot scripts you want to run) is to setup all externals properly - which still has to be done manually in core

$ cd externals $ cat README

and follow the instructions there.

You will also have to enter the password for your bot eventually.

Now you have finished the configuration of core and can continue setting up the jobs to execute.

Setup the webspace
By default, the directory listing on http://tools.wmflabs.org/TOOLNAME is disabled. If you want to allow it for all users, login to your tool account (as already described) and

$ cd ~/public_html $ echo Options +Indexes >> .htaccess

If you run a bot with the  option, you will find the log files within the   directory. If you want to allow users to access it from the web, do

$ cd ~/public_html $ mkdir logs $ cd logs $ ln -s ~/pywikibot-core/logs cor

If you want a specific file type to be handled differently by your browser, e.g.  files like text files, use (confer [https://wikitech.wikimedia.org/w/index.php?title=Help:Move_your_bot_to_Labs&direction=prev&oldid=69904 this])

$ echo AddType text/plain .log >> .htaccess

and (don't forget to) clear your browsers cache afterwards.

Next you might want to consider you  directory

$ cd ~/cgi-bin

follow the hints given at wikitech:Help:Toolforge exactly, e.g. even the two commands

$ /usr/bin/python     # valid $ /usr/bin/env python # in-vali

work and do the same in shell, only the first one is valid and works here, the second is invalid! Another point to mention is that PHP scripts go into public_html, not cgi-bin. Python scripts on the other hand can be placed in public_html or cgi-bin as you wish. It is recommended to use  for documents and keep it listable, whereas   should be used for CGI scripts and be protected (not listable).

Setup the job submission
In order to setup the submission of the jobs you want to execute and use the grid engine you should first consider wikitech:Help:Toolforge and if you are familiar with the Toolserver and its architecture consult Migrating from toolserver also.

In general labs uses Grid Engine and its commands like  et al, this is explained in the Toolforge Job Grid documentation which you should use in order to get an idea which command and what parameters you want to use.

An infinitely running job (e.g. irc-bot) like this ( entry from TS submit host):

becomes

or shorter

the first expression is good for debugging. Memory values smaller than 256MB seem not to work here, since that is the minimum. If you experience problems with your jobs, like e.g. Fatal Python error: Couldn't create autoTLSkey mapping you can try increasing the memory value - which is also needed here, because this script uses a second thread for timing and this thread needs memory too. Therefore use finally

Now in order to create a crontab follow Scheduling jobs at regular intervals with cron and setup for crontab file like: $ crontab -e and enter

Additional configuration
Furthermore additional tools to support you and your bot at work are available:
 * wikitech:Help:Toolforge, basically there is protection from disk failure, but not user managed backups
 * wikitech:Help:Toolforge
 * Gerrit/New repositories
 * Git/New repositories/Requests

Automatic updating git on Toolforge
For automatic updating of pywikibot, you can make an update bash file in your tool's  directory:

and then run crontab -e and enter the following to update your git clone every day at 00:00AM (midnight):

Notice: in these codes yourtoolname is name of your tool account.