Manual:Pywikibot/Installation

Initial setup
There are four basic steps to installing pywikibot:
 * 1) Download or update Python,
 * 2) Download pywikibot,
 * 3) Configure pywikibot's basic settings.

Downloading Python

 * Unix systems: typically have a recent-enough version of Python installed. Check with.


 * Mac OS X 10.8+: has a version of Python that is recent enough to run pywikibot. If you run an older OS X, or are unsure, download and install Python 2.7.10 from http://www.python.org/downloads/


 * Windows: download and install the latest release Python 2.7.10 from http://www.python.org/downloads/


 * Python version: The following versions of Python are supported:

Due to a Unicode bug in the underlying python library, Python 2.7.0 and 2.7.1 are not supported using the Core branch. See also the requirements for a bot, which describes needed python version and pywikibot branches.

Download Pywikibot
The easiest way to download Pywikibot is to use [http://tools.wmflabs.org/pywikibot/ the latest nightly release]. Just download the pywikibot zip file to your computer and decompress the file – there is no further installation required.

Download Pywikibot with Git
For installing with Git you need to run:

With the  option it automatically installs the required submodules. Currently there are two submodules (i18n and spelling); one of them (i18n) is really required even for English language bots:

Download Pywikibot with SVN
If you don't want to use Git you can still use SVN.

Windows users may use the GUI extension TortoiseSVN, see next section below.

For installing with SVN you should run

Download Pywikibot with TortoiseSVN for Windows user
TortoiseSVN is a Windows shell extension with GUI working as SVN client. You may download the current release at http://tortoisesvn.net/downloads.html.

If you like using TortoiseSVN you may use it as follows:


 * 1) Right-click on your preferred directory and execute
 * 2) Choose   and paste the URL
 * 3) Choose   and change the default directory which is   e.g. to   if you like.
 * 4) Confirm with
 * Now we have to install external libraries used by the framework. There are different ways to do that e.g. again with the checkout command. Another way is using properties. You may set it manually but it is very easy to use the preference file named .svnprops coming with the framework which sets all the needed properties:


 * 1) Right-click your working directory
 * 2) Select the last item in the drop list which might be folder's
 * 3) Select   tab and click on   button
 * 4) click   and select the file   in your framework folder
 * 5) Confirm with   for property settings
 * 6) Confirm with   for folder properties
 * 7) Now right-click onto your working copy and select   to download the external libraries.

Right-click on your working copy and choose
 * Updating the working copy

Shortcut in command line
To allow your source code to be developed outside of the pywikibot source directory, add something like:

to a file that gets run on login, usually ~/.bashrc - this avoids typing the export PYTHONPATH part in each time you log in. Naturally, change paths to match your installation.

Similarly, you can set the PYWIKIBOT_DIR environment variable to specify the directory in which user-specific information is stored (in particular, user-config.py which contains login data for the bot).

Updating
The pywikibot framework is a perpetual beta software. Bugfixes, new options, new behavior and changes of the mediawiki software leads to frequently new releases and needs the working copy code to be up to date. Please update your branch regularly (daily period or at least once a week).

Updating nightly dump

 * If you are using a nightly version, the process is a bit more complicated. You have to re-download a full copy from [http://tools.wmflabs.org/pywikibot/ here]. Before installing it, backup your configuration files and scripts (user-config.py, any family file, or custom script that you might have created, and any current dump XML file you're using for a wiki). Replace your pywikibot directory by the new version you just downloaded. Restore your configuration files. If you're not sure of what you're doing, do not erase but keep a backup of your complete old pywikibot directory, to avoid losing any important files.

Updating git

 * If you used Git for downloading Pywikibot, you must go to the your directory and run the following command:
 *  


 * You may need to do   as well, if you need up-to-date i18n files.

Automatic updating git on Wikimedia Labs or Toolserver
For automatic updating you can make update bash file and put it in root and fill it with these commands, For WMF labs (in your service group):

and then run crontab -e and enter the following to make your bot to run every day at 00:00AM (midnight):

Notice: in these cods yourservicegroup is name of your service group (without "-local").

For auto-updating in toolserver you just need to as the same as above but instead of "/data/project/yourservicegroup/" you need to enter "/data/project/yourusername/" which is "yourusername" is your user name!

Updating with TortoiseSVN (for Windows user)
Updating the working copy with TortoiseSVN is just easy: Right-click on your working copy directory and choose

Dependencies
Use  package management system to install dependencies.

It is included by Python 2.7.9 and Python 3.4. To install all dependencies just run the following command:

For a single package use:

To uninstall a single package use:

Setup on Wikimedia Labs/Tool Labs server
In order to install your bot onto the Wikimedia servers and run it from there, make sure first to become familiar with Wikimedia Labs/Tool Labs environment.

In the next step you have to request several accounts (for labs, for the tools project, your tool), provide an ssh key and so on. How to do this and then proceed, is described in full detail in Setup pywikibot on Labs.

Pywikibot source repo moved (from svn) to git, please confer Manual:Pywikipediabot/Gerrit first.

The bots projects here has become obsolete use tools now, in order to do so follow Tools/Help to get an account. Then create your tool (service group).

If you used the toolserver in the past and know how everything used to work there, confer migrating from toolserver for more info.

Now you are ready to start. Login to Labs tools project: $ ssh USERNAME@tools-login.wmflabs.org

switch to the tool account with maintainer@tools-login:~$ become toolname local-toolname@tools-login:~$

Now install/clone the pywikibot code to your tool account as described below.

Install the bot code
Similar to the instructions given in [http://lists.wikimedia.org/pipermail/pywikipedia-l/2013-August/008168.html this mail] do: $ git clone --recursive https://gerrit.wikimedia.org/r/pywikibot/core.git pywikibot-core $ cd pywikibot-core

Now you have to setup pywikibot. Choose any one of the following processes to configure your system:
 * Execute
 * Run your favorite bot script (e.g. ) since you are doing this in a fresh clone, it will trigger a bunch of questions on how you want to configure your local copy, answer them carefully in order to proceed.
 * If you already have a config file(s) from a previous version, you can copy those existing files into the right places (e.g. pywikibot-compat/).

Further things you might have to do (depending on what bot scripts you want to run) is to setup all externals properly - which still has to be done manually in core $ cd externals $ cat README

and follow the instructions there.

You will also have to enter the password for your bot eventually.

Now you have finished the configuration of core and can continue setting up the jobs to execute.

Setup the webspace
Per default, the directory listing on http://tools.wmflabs.org/TOOLNAME is disabled. If you want to allow it for all users, login to your tool account (as already described) and $ cd ~/public_html $ echo Options +Indexes >> .htaccess

If you run a bot with the  option, you will find the log files within the logs/ directory. If you want to allow users to access it from the web, do $ cd ~/public_html $ mkdir logs $ cd logs $ ln -s ~/pywikibot-core/logs cor

If you want a specific file type to be handled differently by your browser, e.g. .log files like text files, use (confer [<tvar|Move_to_Labs_URL>https://wikitech.wikimedia.org/w/index.php?title=Help:Move_your_bot_to_Labs&direction=prev&oldid=69904</> this]) $ echo AddType text/plain .log >> .htaccess

and (don't forget to) clear your browsers cache afterwards.

Next you might want to consider you cgi-bin</tt> directory $ cd ~/cgi-bin

follow the hints given at wikitech:Nova Resource:Tools/Help exactly, e.g. even the two commands

$ /usr/bin/python     # valid $ /usr/bin/env python # in-vali

work and do the same in shell, only the first one is valid and works here, the second is invalid! Another point to mention is that PHP scripts go into public_html, not cgi-bin. Python scripts on the other hand can be placed in public_html or cgi-bin as you wish. I would recommend to use public_html</tt> for documents and keep it listable, whereas cgi-bin</tt> should be used for CGI scripts and be protected (not listable).

Setup the job submission
In order to setup the submission of the jobs you want to execute and use the grid engine you should first consider wikitech:Nova Resource:Tools/Help and if you are familiar with the Toolserver and its architecture consult Migrating from toolserver also.

In general labs uses SGE and its commands like qsub</tt> et al, this is explained in this document which you should use in order to get an idea which command and what parameters you want to use.

An infinitely running job (e.g. irc-bot) like this (cronie</tt> entry from TS submit host):

becomes

or shorter

the first expression is good for debugging. Memory values smaller than 256MB seam not to work here, since that is the minimum. If you experience problems with your jobs, like e.g. Fatal Python error: Couldn't create autoTLSkey mapping you can try increasing the memory value - which is also needed here, because this script uses a second thread for timing and this thread needs memory too. Therefore use finally

Now in order to create a crontab follow Scheduling jobs at regular intervals with cron and setup for crontab file like: $ crontab -e and enter

Additional configuration
Furthermore additional tools to support you and your bot at work are available:
 * wikitech:Nova Resource:Tools/Help, basically out-of-the-box but just for a short time period
 * wikitech:Nova Resource:Tools/Help
 * Gerrit/New repositories
 * Git/New repositories/Requests

Creating user files
As a final step before using the bot scripts, you have to create user configuration files. The main one is. If you execute your favorite script after installation (for example, ) then pywikibot ask questions on how you want to configure your local copy and will create these files. You can also run generate user files.py to create these files.

Running Pywikibot under Windows
Please notice the instruction at Manual:Pywikibot/Windows.

Verifying your installation
With the core branch, entering in the command line python pwb.py login should prompt for the login password for your bot, and then report Logged in on somewiki:lang as bot username.