User:Kmenger/ToolLabsGuide

What is Tool Labs
Tool Labs is a reliable, scalable hosting environment for community developers working on tools and bots that help users maintain and use wikis. The cloud-based infrastructure was developed by the Wikimedia Foundation and is supported by a dedicated group of Wikimedia Foundation staff and volunteers.

Tool Labs is a part of the Labs project, which is designed to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure, including MediaWiki, and to do analytics and  bot work.

Rationale
Tool Labs was developed in response to the need to support external tools and their developers and maintainers. The system is designed to make it easy for maintainers to share responsibility for their tools and bots, which helps ensure that no useful tool gets ‘orphaned’ when one person needs a break. The system is designed to be reliable, scalable and simple to use, so that developers can hit the ground and start coding.

Features
In addition to providing a well supported hosting environment, Tool Labs provides:
 * support for Web services, continuous bots, and scheduled tasks
 * access to replicated production databases
 * easily shared management of tool accounts, where tools and bots are stored
 * a grid engine for dispatching jobs
 * support for mosh, SSH, SFTP without complicated proxy setup
 * time-travel backups for short-term data recovery
 * version control via Gerrit and Git
 * support for Redis

Architecture and terminology
Tool Labs has essentially four components: the bastion hosts, the grid, the web cluster, and the databases. Users access the system via one of two Tool Lab projects: ‘tools’ or ‘toolsbeta’. To request an account on the ‘tools’ project, where most tool and bot development is hosted and maintained, please see Tools Access Request.

Bastion hosts, grid, web cluster, databases
The four main components of Tool Labs, in a nutshell:

Bastion hosts

The bastion host is where users log in to Tool Labs. Currently, Tool Labs has two bastion hosts:


 * tools-login.wmflabs.org


 * tools-dev.wmflabs.org

The two hosts are functionally identical, but we request that heavy processing (compiles, etc) be done only on tools-dev.wmflabs.org to keep interactive performance on tools-login.wmflabs.org snappy.

The grid

The Tool Labs grid, implemented with Open Grid Engine (the open-source fork of Sun Grid Engine) permits users to submit jobs from either a log-in account on the bastion host or from a Web service. Submitted jobs are added to a work queue, and the system finds a host to execute them. Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once. If a continuous job fails, the grid will automatically restart the job so that it keeps going.For more information about the grid, please see Submitting, managing and scheduling jobs on the grid.

The Web cluster

The Tool Labs Web cluster is fronted by a Web proxy, which supports SSL and is open to the Internet. Any of the servers in the cluster can serve any of the hosted Web tools as Tool Labs uses a shared storage system; the proxy distributes between the Web servers. The cluster uses suPHP to run scripts and CGI, and will soon support WSGI. Note that individual tool accounts have both a  ~/public_html/  and a ~/cgi-bin/ directory in the home directory for storing Web files. For more information, please see Web services.

The databases 

Tool Labs supports two sets of databases: the production replicas and user-created databases, which are used by individual tools. The production replicas follow the same setup as production, and the information that can be accessed from them is the same as that which normal registered users (i.e.: not  +sysop or other types of advanced permissions) can access on-wiki or via the API. Note that some data has been removed from the replicas for privacy reasons.User-created databases can be created by either a user or a tool on the replica servers or on a local ‘tools’ project database.

Projects: Tools and Toolsbeta
Like the rest of Labs, Tool Labs is organized into ‘projects’. Currently, Tool Labs consists of two projects: ‘tools’ and ‘toolsbeta’, which are described in more detail here: The ‘tools’ project is where tools and bots are developed and maintained. ‘toolsbeta’ is used for experiments in the Tool Labs environment itself--things like new systems or experimental versions of system libraries that could affect other users. In general, every tool maintainer should work primarily on the "tools" project, only doing work on toolsbeta when changes to Tool Labs itself need to be tested to support their tool.
 * tools  project: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools
 * toolsbeta  project: https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta

Instances
Developers working in Tool Labs do not have to create or set up virtual machines (i.e., Lab ‘instances') as the Tool Lab project admins create and manage them. The term will come up in the Labs documentation; otherwise, don’t worry about it.

Tool Labs policies
All tools and bots developed and maintained on Tool Labs must adhere to the terms of use that will be available here when they are finalized:


 * Tool Labs > Rules

Specifically, tools must be Private information must be handled carefully, if at all. Note that private user information has been redacted from the replicated databases provided by the system.
 * Open source
 * Open data

As the Tool Labs environment is shared, we ask that you strive not to break things for others, and to be considerate when using system resources.

Individual wiki policies (these differ!)
When developing on Tool Labs, please adhere to the bot policies of the wikis your bot interacts with. Each wiki has its own guidelines and procedures for obtaining approval. The English Wikipedia, for example, requires that a bot be approved by the Bot Approvals Group before it is deployed, and that the bot account be marked with a ‘bot’ flag. See |Wikipedia Bot policy for more information on the English Wikipedia.

For general information and guidelines, please see Bot policy.

Contact
We’d love to hear from you! You can find us here:


 * On IRC: #wikimedia-labs on Freenode A great place to ask questions, get help, and meet other Tool Lab developers. See Help:IRC for more information.


 * Via mailing list: Labs-l@lists.wikimedia.org A list for announcements and discussion related to the Wikimedia Labs project. You can find the archives here: http://lists.wikimedia.org/pipermail/labs-l/
 * Found a bug?: Bugs can be posted to Bugzilla: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs

Getting access to Tool Labs
Anyone can view the source code and the output of most tools and bots, and anyone can get an account of their own as well.

To access Tool Labs you need:


 * to create a Labs account, which provides shell access (you must upload an SSH key)
 * to request access to the 'tools' project

Steps for creating a Labs account, creating and uploading an SSH key, and for requesting access to the 'tools project' are described in the next sections.

Creating a Labs account on Wikitech
Before you can access Tool Labs, you must create a Labs account on Wikitech, which is the general interface for everything Labs.

Sign up for a Labs account here: Request account (you will be asked to enter the new account's information)

The "Instance shell account name" you specify in the Create Account form will be your Unix username on all Labs projects. If you forget your username, you can always find it under Preferences > Instance shell account name.

Once you have created a Labs account you will be added to a list of users to be approved for shell access, which you can see here: Shell Access Requests.

Generating and uploading an SSH key
In order to access Labs servers using SSH, you must provide a public SSH key. Once you have created a Labs account, you can specify a public key on the 'OpenStack' tab of your Wikitech preferences.

Specify the SSH key here: |OpenStack Preferences

Generating a key in Windows
To generate an SSH key in Windows:


 * 1)     Open PuttyGen
 * 2)     Select an SSH-2 RSA key
 * 3)     Click the Generate button
 * 4)     Move your mouse around until the progress bar is full
 * 5)     Type in a passphrase (you will need to remember this) and confirm it
 * 6)     Save the private key and public key onto your local machine
 * 7)     From the text field 'Public key for pasting into OpenSSH authorized_keys file' right click and copy
 * 8)     Insert this into your 'OpenStack' tab of your Wikitech preferences

Generating a key in Linux
Modern Unix systems include the OpenSSH client (if not then install it). To generate a key, use:

ssh-keygen -t rsa

This will store your private key in $HOME/.ssh/id_rsa, and your public key in $HOME/.ssh/id_rsa.pub. You can use different filenames (with -f parameter), but these are the default filenames, so it's easiest to not change them.

Requesting access to the 'tools' project
Once you have created a Labs account, you must request access to the ‘tools’ project by submitting a Tools Access Request.

Submit a request here: Tools Access Request

Requests for access are generally dealt with within the day (often faster), though response-time may be longer depending on admin availability. If you need immediate assistance, please contact us on IRC.

Receiving access to the 'tools' project
Once your 'tools' project access request has been processed, you will become a member of the 'tools' project, and will be able to access it using the "Instance shell account name" provided when creating your Labs account and the private key matching the public key you supplied for authentication. For more information about accessing the project, please see Accessing Tool Labs.

Notification
You will be notified on Wikitech that your user rights were changed, that your request was linked from 'Nova Resource:Tools', and that you have been added to the project Nova Resource:Tools. You will also receive email explaining that your user rights have been changed, and that you are now a member of the group 'shell'. In other words, your Tool Labs account is ready for you to use!

Storage and use
Although you access Tool Labs via your Labs account,  we strongly recommend against saving data or tools in any space  that is accessible to individuals only. Tools and bots should be maintained in Tool accounts, which have flexible memberships (i.e., multiple people can help maintain the code!). For more information about Tool accounts, please see Joining and creating a Tool account.

Accessing Tool Labs
Tool Labs can be accessed in a variety of ways--from its public IP to a GUI client. Please see Help:Access (https://wikitech.wikimedia.org/wiki/Help:Access ) for general information about accessing Labs. Pointers to more information on specific means of access below.

Tools home page
The Tools home page:http://tools.wmflabs.org/

The Tools home page is publicly available and contains a list of all currently hosted Tool accounts along with the name(s) of the maintainers for each. Individual tool accounts that have an associated web page will appear as links. Users with access to the 'tools' project can create new tool accounts here, and add or remove maintainers to and from existing tool accounts.

SSH/SFTP/SCP
Users can SSH to the 'tools' project via its bastion host: tools-login.wmflabs.org, provided that a public SSH key has been uploaded to the Labs account.

ssh yourshellaccountname@tools-login.wmflabs.org

Note that if you plan to do heavy processing (compiling, etc), you should SSH to tools-dev.wmflabs.org.

Using 'take' to transfer ownership of uploaded files
Once you have logged in via SSH, you can transfer files via sftp and scp. Note that the transferred files will be owned by you. You will likely wish to transfer ownershihp to your tool account. To do this:

1. Become your tool account using 'become':

maintainer@tools-login:~$ become toolaccount local-toolaccount@tools-login:~$

2. As your tool account, 'take' ownership of the files:

local-toolaccount@tools-login:~$ take FILE

The 'take' command will change the ownership of the file(s) and directories recursively to the calling user (in this case, the tool account).

Using multiple ssh agents
If you use multiple ssh-agents (to connect to your personal or company system, for example), see Managing Multiple SSH Agents for more information about setting up a primary and a Labs agent.

Putty and WinSCP
Note that instructions for accessing Tool Labs with Putty and WinSCP differ from the instructions for using them with other Labs projects. Please see Help> Access to ToolLabs instances with PuTTY and WinSCP for information specific to Tool Labs.

Other graphical file managers (e.g., Gnome/KDE)
For information about using a graphical file manager (e.g., Gnome/KDE), please see Accessing Tool Labs > Accessing instances with a graphical file manager

What is a Tool account?
Tool accounts, which can be created by any ‘tools’ project member, are fundamental to the structure and organization of Tool Labs. Although each tool account has a user ID, they are not personal accounts (like a Labs account), rather services that consist of a user and group ID (i.e., a unix uid-gid pair) that are intended to run the actual tool or bot.


 * Unix user: local-toolname
 * Unix group: local-toolname

Members of the Unix group include:


 * the tool account creator
 * the tool account itself
 * (optionally, but encouraged!) additional tool maintainers

Maintainers may have more than one tool account, and tool accounts may have more than one maintainer. Every member of the group has the authorization to sudo to the tool account. By default, only members of the group have access to tool account's code and data.

A simple way for maintainers to switch to the tool account is with ‘become’:

maintainer@tools-login:~$ become toolname local-toolname@tools-login:~$

In addition to the user/group pair, each tool account includes:


 * A home directory on shared storage: /data/project/toolname
 * A ~/public_html/ and ~/cgi-bin/ directory, which are visible at http://tools.wmflabs.org/toolname/  and http://tools.wmflabs.org/toolname/cgi-bin, respectively
 * Database access credentials: replica.my.cnf, which provide access to the production database replicas as well as to project-local databases.
 * Access to the continuous and task queues of the compute grid

Joining an existing Tool account
All tool accounts hosted in Tool Labs are listed on the Tools home page. If you would like to be added to an existing account, you must contact the maintainer(s) directly.

If you would like to add (or remove) maintainers to a tool account that you manage, you may do so with the 'add' link found beneath the tool name on the Tools home page.

Creating a new Tool account
Members of the ‘tools’ project can create tool accounts from the Tools home page:


 * 1) Navigate to the Tools home page: http://tools.wmflabs.org/
 * 2) Select the “create new tool” link (found beside “Hosted tools” near the top of the page
 * 3)  Enter a  “Service group name”. The service group name will be used as the name of your tool account.

Do not prefix your service group name with local-. The management interface will do so automatically where appropriate, and there is a known issue that will cause the account to be created improperly if you do.

Note: If you have only recently been added to the ‘tools’ project, you may get an error about not having appropriate credentials. Simply log out and back in to Wikitech to fix this

The tool account will be created and you will be granted access to it within a minute or two. If you were already logged in to your Labs account through SSH, you will have to log off then back in before you can access the tool account.

Deleting a Tool account
You can't delete a tool account yourself, though you can delete the content of your directories. If you really want a tool account to be deleted, please contact an admin.

Using Toolsbeta
Nearly all tool development is done on the 'tools' project, and 99.9% of the time, creating a tool account on this project will serve your needs. However, if your tool or bot requires an experimental library or a significant change to the 'tools' infrastructure--anything that could potentially negatively impact existing tools--you should experiment with the new infrastructure on toolsbeta. To request access to toolsbeta, please visit #wikimedia-labs on IRC. You can also request access via the labs-l mailing list or via Bugzilla.

Customizing a Tool account
Once you have created a tool account, there are a few things that you can customize to make the tool more easily understood and used by other users. These include: Tool Labs will soon support mail to both Labs users and tool accounts (mail to a tool account will go to all maintainers by default). You can customize mail settings as well.
 * adding a tool account description (the description will appear on the Tools home page beside the tool name)
 * creating a home page for your tool (if you create a home page for the tool, it will be linked from the Tools home page automatically)

Creating a tool web page
To create a web page for your tool account, simply place an index.html file in the tool account's ~/public_html/ directory. The page can be a simple description of the tool or bot with basic information on how to set it up or shut it down, or it contain an interface for the web service. To see examples of existing tool web pages, click any of the linked Tool names on the Tools home page.

Note that some files, such as PHP files, will give a 500 error unless the owner of the file is tool account.

Creating a tool description
To create a tool description:

1.    Log into your Labs account and become your tool account:

maintainer@tools-login:~$ become toolname

2.    Create a ‘.description’ file in the tool account’s home directory. Note that this file must be HTML:

local-toolname@tools-login:~$ vim .description

3.    Add a brief description (no more than 25 words or so) and save the file.

4.    Navigate to the Tools home page. Your tool account description should now appear beside your tool account name.

Configuring mail -- mail forwarding
Mail from system daemons (grid, cron, etc.) is delivered to tool and user accounts. By default, tool accounts forward their mail to their maintainers' accounts, while user accounts store mail locally and users can read it (e.g., with mail).

To forward mail to your personal mail address from a Labs account:

1. Log in to your Labs account

2. In your home directory, create a file ‘.forward’

maintainer@tools-login:~$ vim .forward

3. Add the forwarding email address on a single line, e.g.

me@example.invalid

4. Ensure that .forward is only writable only by you, the account user. If ‘.forward’ is writable by anyone other than you, mail is not delivered at all!

maintainer@tools-login:~$ chmod 600 ~/.forward

You can also use a .forward file in a tools account to redirect mail to a specified address (e.g., a mailing list) instead of sending all messages to the individual maintainers (which is the default).

... any other things people would like to use/configure?... 5.4   Configuring bots -- running them

In order to run a bot first of all you need to configure it. At first you should become your tool and after that you can command:

git clone --depth 3 https://gerrit.wikimedia.org/r/pywikibot/core.git cd core git submodule update --init cd externals git clone https://gerrit.wikimedia.org/r/pywikibot/spelling.git

you can use "compat" (formerly "trunk") instead of "core" (formerly "rewrite") if you're more familliar with trunk branch: git clone https://gerrit.wikimedia.org/r/pywikibot/compat.git pywikipedia cd pywikipedia git submodule update --init cd externals git clone https://gerrit.wikimedia.org/r/pywikibot/spelling.git

installing your bot in core or compat has two diffrenet methods, for core command python setup.py and answer the questions, for compat, command: python login.py and answer the questions. After installing you can run your codes directly via shell access but this is highly discouraged you must run your codes via jsub, cron, or SGE. jsub will use GRID engine (see above) For submitting a job you must command: jsub -once -N YOURJOBNAME python /data/project/YOUR-TOOL/PATHOFYOURCODE for example for running welcome.py in jsub you should command: jsub -once -N welcome python /data/project/YOUR-TOOL/core/scripts/welcome.py if you're using compat: jsub -once -N welcome python /data/project/YOUR-TOOL/pywikipedia/welcome.py

If you want to see status of your job command: qstat and if you want to see output of your job you can see them via: vim YOURJOBNAME.err vim YOURJOBNAME.out the former shows errors and the latter shows outputs for deleting a job, command: qdel NUMBEROFJOB you can find numberofjob in qstat. For further information see below

For further information about running bots you can see this help

Setting up code review and version control
Although it's possible to just stick your code in the directory and mess with it manually every time you want to change something, your future self and your future collaborators will thank you if you instead use source control, a.k.a. version control and a code review tool. Wikimedia Labs makes it pretty easy to use Git for source control and Gerrit for code review, but you also have other options.

Gerrit/Git
Access to Git is managed via Wikimedia Labs and integrated with Gerrit. In order to use them for code review and version control with your tool accounts, you must request access. For more information, please see Gerrit/New repositories https://www.mediawiki.org/wiki/Gerrit/New_repositories

For more information about using Git and Gerrit in general, please see Gerrit.

Database access
Tool and Labs accounts are granted access to replicas of the production databases. Private user data has been redacted from these replicas (some rows are elided and/or some columns are made NULL depending on the table), but otherwise the schema is, for all practical purposes, identical to the production databases, and are sharded into clusters in much the same way.

Database credentials (user name/password) are stored in the 'replica.my.cnf' file found in the tool account’s home directory. To use these credentials with command-line tools by default, copy 'replica.my.cnf' to '.my.cnf'.

Naming conventions
As a convenience, each mediawiki project database (enwiki, bgwiki, etc) has an alias to the server it is hosted on. The alias has the form:


 * project.labsdb

where 'project' is the name of a hosted mediawiki project (enwiki bgwiki bgwiktionary cswiki enwikiquote enwiktionary eowiki fiwiki idwiki itwiki nlwiki nowiki plwiki ptwiki svwiki thwiki trwiki zhwiki commonswiki dewiki wikidatawiki arwiki eswiki... for a complete list, look at the /etc/hosts file on tools-login).

The database names themselves consist of the mediawiki project name, suffixed with _p (an underscore, and a p), for example:


 * enwiki_p (for the English Wikipedia replica)

Connecting to the database replicas
You can connect to the database replicas by specifying access credentials and the host of the replicated database. For example:

To connect to the English Wikipedia replica: mysql --defaults-file="${HOME}"/replica.my.cnf -h enwiki.labsdb enwiki_p

To connect to Wikidata: mysql --defaults-file=~/replica.my.cnf -h wikidatawiki.labsdb

To connect to Commons: mysql --defaults-file=~/replica.my.cnf -h commonswiki.labsdb

There is also a shortcut for connecting to the replicas: sql [_p]    [MAP: _p is optional, but implicit (i.e. the sql tool will add it if absent)]

To connect to the English Wikipedia database replica using the shortcut, simply type:

sql enwiki

Creating new databases
User-created databases can be created on the database hosting the replica servers or on a database local to the 'tools' project: tools-db. The latter tends to be a bit faster since that server has less heavy activity, and tools-db is the recommended location for user-created databases when no interaction with the production replicas is needed. Users have all privileges on the created database and grant options.

Database names must start with the name of the credential user, which can be found in your ~/replica.my.cnf file (the name looks something like 'p50252g21636'), followed by two underscores and then the name of the database: 'username__DBName'

Note that users are granted complete control over there username__, but nothing else.

Steps to create a user database on the replica servers
If you would like your database to interact with the replica databases (i.e., if you need to do actual SQL joins with the replicas, which can only be done on the same cluster) you can create a database on the replica servers.

To create a database on the replica servers:

1. Connect to the replica servers with the replica.my.cnf credentials. You must specify the host of the replica (e.g., enwiki.labsdb):

mysql --defaults-file="${HOME}"/replica.my.cnf -h xxwiki.labsdb

2. In the mysql console, create a new database (where USERNAME is your credentials user and DBNAME the name you want to give to your database):

MariaDB [(none)]> CREATE DATABASE USERNAME__DBNAME

You can then connect to your database using: mysql --defaults-file="${HOME}"/replica.my.cnf -h xxwiki.labsdb USERNAME__DBBAME

Steps to create a user database on tools-db
To create a database on tools-db:

1. Connect to tools-db with the replica.my.cnf credentials:

mysql --defaults-file="${HOME}"/replica.my.cnf -h tools-db

2. In the mysql console, create a new database (where USERNAME is your credentials user and DBNAME the name you want to give to your database):

MariaDB [(none)]> CREATE DATABASE USERNAME__DBNAME

You can then connect to your database using: mysql --defaults-file="${HOME}"/replica.my.cnf -h tools-db USERNAME__DBBAME

Joins between commons and wikidata and other project databases
??????? -- can you help with this section---???? <<<<<<<<<<<<<<<<<<            That needs to be written from scratch. I'll sit down and look at it tonight.