User:Kmenger/ToolLabsSprint

Plan: July 23-25, a bunch of us are going to work on documentation, here on Etherpad, on the wikitech.wikimedia.org wiki, and in #wikimedia-labs on Freenode. Join us :)

Participants include: Kirsten Silke (part of the time) Sumana Ryan Coren Johannes (don't have lots of time, but would like to add some documentation about catgraph) Addshore (don't have lots of time, but will try to keep track of what happening and participate when and where i can) [=

(add yourself)

This document contains: 1. An initial stab at an outline for a Tool Labs guide, along with source links--to draw from or point to--for each section 2. Additions to the "Migrating of Toolserver guide", with source links 3. An inventory of current documentation relating to Tool Labs, along with a proposal for a) updating b) redirection c) Appears current

All help much appreciated!

I also have some questions/notes here: http://www.mediawiki.org/wiki/User:Kmenger/Tool_Labs



Tool Labs guide

1   What is Tool Labs

Tool Labs is a reliable, scalable hosting environment for community developers working on tools and bots that help users maintain and use wikis. The cloud-based infrastructure was developed by the Wikimedia Foundation and is supported by a dedicated group of Wikimedia Foundation staff and volunteers.

Tool Labs is a part of the Labs project, which is designed to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure, including MediaWiki, and to do analytics and bot work.

1.1   Rationale

Tool Labs was developed in response to the need to support external tools and their developers and maintainers. The system is designed to make it easy for maintainers to share responsibility for their tools and bots, which helps ensure that no useful tool gets ‘orphaned’ when one person needs a break. The system is designed to be reliable, scalable and simple to use, so that developers can hit the ground and start coding.

1.2   Features

In addition to providing a well supported hosting environment, Tool Labs provides:

·      support for Web services, continuous bots, and scheduled tasks ·      access to replicated production databases ·      easily shared management of tool accounts, where tools and bots are stored ·      a grid engine for dispatching jobs ·      support for mosh, SSH, SFTP without complicated proxy setup ·      time-travel backups for short-term data recovery ·      Ganglia, Icinga, and Nagios systems to systems to help monitor tools ·      Version control via Gerrit and Git

·      1.3    Architecture and terminology

Tool Labs has essentially four components: the bastion hosts, the grid, the web cluster, and the databases. Users access the system via one of two Tool Lab projects: ‘tools’ or ‘toolsbeta’. To request an account on the ‘tools’ project, where most tool and bot development is hosted and maintained, please see Special:FormEdit/Tools Access Request

1.3.1 Bastion hosts, grid, web cluster, databases

Bastion hosts The bastion host is where users log in to Tool Labs. Currently, Tool Labs has two bastion hosts:

tools-login.wmflabs.org

tools-dev.wmflabs.org

The two hosts are functionally identical, but we request that heavy processing (compiles, etc) be done only on tools-dev.wmflabs.org to keep interactive performance on tools-login.wmflabs.org snappy.

The grid The Tool Labs grid, implemented with Open Grid Engine (the open-source fork of Sun Grid Engine) permits users to submit jobs from either a log-in account on the bastion host or from a Web service. Submitted jobs are added to a work queue, and the system finds a host to execute them. Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once. If a continuous job fails, the grid will automatically restart the job so that it keeps going.

For more information about the grid, please see [#Submitting, managing and scheduling Tool jobs on the grid]

The Web cluster

The Tool Labs Web cluster is fronted by a Web proxy, which supports SSL and is open to the Internet. Any of the servers in the cluster can serve any of the hosted Web tools as Tool Labs uses a shared storage system; the proxy distributes between the Web servers. The cluster uses suPHP (www.suphp.org/) to run scripts and CGI, and will soon support WSGI (www.wsgi.org/‎).

Note that individual tool accounts have both a ~/public_html/  and a ~/cgi-bin/ directory in the home directory for storing Web files. For more information, please see [#Web services]

The databases Tool Labs supports two sets of databases: the production replicas and user-created databases, which are used by individual tools.

The production replicas follow the same setup as production, and the information that can be accessed from them is the same as that which normal registered users (i.e.: not +sysop or other types of advanced permissions) can access on-wiki or via the API. Note that some data has been removed from the replicas for privacy reasons.

User-created databases can be created by either a user or a tool on the replica servers or on a local ‘tools’ project database.

1.3.2 Projects: Tools and Toolsbeta

Like the rest of Labs, Tool Labs is organized into ‘projects’. Currently, Tool Labs consists of two projects: ‘tools’ and ‘toolsbeta’, which are described in more detail here: Tools project: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools

Toolsbeta project: https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta

The ‘tools’ project is where tools and bots are developed and maintained. ‘Toolsbeta’ is used for experiments in the Tool Labs environment itself--things like new systems or experimental versions of system libraries that could affect other users. In general, every tool maintainer should work primarily on the "tools" project, only doing work on toolsbeta when changes to Tool Labs itself need to be tested to support their tool.

1.3.3 Instances: Tool accounts

Developers working in Tool Labs do not have to create or set up Lab ‘instances,’ as the system creates them automatically: Each Tool account is an instance. The term will come up in the Labs documentation; otherwise, don’t worry about it.

1.4   Rules of use

1.4.1 Tool Labs policies

All tools and bots developed and maintained on Tool Labs must adhere to the terms of use that will be available here when they are finalized: Tool Labs > Rules (https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Rules)

Specifically, tools must be ·      Open source ·      Open data Private information must be handled carefully, if at all. Note that private user information has been redacted from the replicated databases provided by the system.

As the Tool Labs environment is shared, we ask that you strive not to break things for others, and to be considerate when using system resources.

For more information about Labs policies, please see Wikimedia Labs > Terms of use ( https://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use)

1.4.2 Individual wiki policies (these differ!)

When developing on Tool Labs, please adhere to the bot policies of the wikis your bot interacts with. Each wiki has its own guidelines and procedures for obtaining approval. The English Wikipedia, for example, requires that a bot be approved by the Bot Approvals Group before it is deployed, and that the bot account be marked with a ‘bot’ flag..

For general information and guidelines, please see ‘Bot policy’

1.5   Contact

We’d love to hear from you! You can find us here:

On IRC: #wikimedia-labs on Freenode A great place to ask questions, get help, and meet other Tool Lab developers. See Help:IRC for more information. https://wikitech.wikimedia.org/wiki/Help:IRC

Via mailing list: Labs-l@lists.wikimedia.org A list for announcements and discussion related to the Wikimedia Labs project. You can find the archives here: http://lists.wikimedia.org/pipermail/labs-l/

Found a bug?: Bugs can be posted to Bugzilla: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs

> Sources: :http://www.mediawiki.org/wiki/File:Tool_Labs_presentation_(Hackaton_2013).pdf https://wikitech.wikimedia.org/wiki/Help:Terminology http://meta.wikimedia.org/wiki/Bot http://en.wikipedia.org/wiki/Wikipedia:Bot_Approvals_Group https://meta.wikimedia.org/wiki/Bots https://meta.wikimedia.org/wiki/Bot_policy https://www.mediawiki.org/wiki/Wikimedia_Labs https://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005332.html http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Languages http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Projects (tools/toolsbeta info) https://wikitech.wikimedia.org/wiki/Help:IRC https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs (bugzilla)

2   Getting access to Tool Labs Anyone can view the source code and the output of most tools and bots, and anyone can get an account of their own as well. 2.1   Creating an account on Wikitech 2.2   Generating and setting your SSH key 2.2.1   Generating a key in Windows 2.2.2   Generating a key in Linux 2.3   Requesting a Labs account 2.4 Receiving a Labs account 2.4.1 Notification 2.4.2 Storage and use >Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Getting_access https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_instances (advice about SSH keys) https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents (advice about multiple SSH keys) https://www.mediawiki.org/wiki/Wikimedia_Labs/Account_creation_text https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Create_a_User_Account https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Home_directory_storage http://lists.wikimedia.org/pipermail/toolserver-l/2013-April/005952.html (re 2.4.1: note to clarify notification/initial experience) https://wikitech.wikimedia.org/wiki/Help:Shared_storage (not using indiv. account for shared storage)

3   Accessing Tool Labs 3.1   Tools home page 3.2   SSH access 3.3   sftp access 3.4   Using Putty and WinSCP 3.5   Using a graphical file manager 3.6    HTTP access >Sources: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_instances https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_a_graphical_file_manager http://www.mail-archive.com/labs-l@lists.wikimedia.org/msg00375.html for Gnome/KDE users and https://wikitech.wikimedia.org/wiki/User:Wikinaut/Help:Access_to_instances_with_PuTTY_and_WinSCP for Windows users https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Access_Project_Instances https://wikitech.wikimedia.org/wiki/Help:Access https://wikitech.wikimedia.org/wiki/Help:Putty https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#access_to_instances:_http.2Fsftp https://wiki.toolserver.org/view/Transferring_files (toolserver docs on transferring files)

4   Joining and creating a Tool account 4.1 What is a Tool account? 4.2 Joining an existing Tool account 4.3 Creating a new Tool account 4.3.1 What comes with a new Tool account? 4.3.1.1 Home directory 4.3.1.1 Published directories ~/public_html/ and ~/cgi-bin 4.3.1.2 Database access credentials (replica.my.cnf) 4.3.2 Storage limits 4.4 Deleting and renaming a Tool account 4.5 Using Toolsbeta 4.5.1 When to use Toolsbeta 4.5.2 Accessing Toolsbeta Coren> kma500: IRC should be the first port of call since it tends to be fastest/easiest. Followed by discussion on labs-l or even just a bugzilla. Honestly, I don't expect end users to want to hop to toolsbeta unless suggested by one of us ("My tool doesn't work with Foo version 1.0 -- let's try to see if we can backport 2.0 on toolsbeta") >Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Your_tool_account https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Create_a_new_tool https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Project_storage https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Join_Projects https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools [note time lag btw tool account creation and user becomming tool] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Published_directories https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Can_I_have_a_subdomain_for_my_web_service.3F https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#Other_implementation_details [note about requesting tool resources] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools#Can_I_delete_a_tool.3F https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuff (on bot credentials/multiple maintainer tool accounts) http://lists.wikimedia.org/pipermail/labs-l/2013-June/001311.html (info on toolsbeta 'bots') http://lists.wikimedia.org/pipermail/labs-l/2013-June/001313.html (more on toolsbeta) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001194.html (more on toolsbeta) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001198.html (more on toolsbeta) https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta http://lists.wikimedia.org/pipermail/labs-l/2013-March/001018.html (re: new package install)

5   Customizing your Tool account 5.1   Creating a tool description and web page 5.2   Configuring mail 5.3   A few notes about permissions ... any other things people would like to use/configure?... > Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Tool_Labs.27s_landing_page https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Mail http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Mail:_to_do https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Tool_accounts https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools#What_about_file_permissions.3F_Who_can_see_my_code.3F http://lists.wikimedia.org/pipermail/labs-l/2013-May/001177.html & http://lists.wikimedia.org/pipermail/labs-l/2013-May/001179.html (note about permission) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001181.html (note about permission for public_html) http://lists.wikimedia.org/pipermail/labs-l/2013-February/000886.html (idea for what to put in html page)

6.   Setting up code review and version control Although it's possible to just stick your code in the directory and mess with it manually every time you want to change something, your future self and your future collaborators will thank you if you instead use source control, a.k.a. version control and a code review tool. Wikimedia Labs makes it pretty easy to use Git for source control and Gerrit for code review, but you also have other options. 6.1 Gerrit/Git 6.2. Other options > Sources: https://www.mediawiki.org/wiki/Gerrit/New_repositories (gerrit) https://www.mediawiki.org/wiki/Gerrit https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver (other versioning options) https://wikitech.wikimedia.org/wiki/Help:Getting_Started#to_Wikimedia_Code_Review https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Git_committers

7   Database access 7.1   Database replicas and naming conventions 7.2   Connecting to the replicated databases 7.3   Creating databases 7.3.1   Steps to create a user database on tools-db 7.3   Joins between commons and wikidata and other project databases > Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features (status of join) http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005382.html http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Database_replication.2Faccess https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Create_new_databases https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Replicated_databases https://wikitech.wikimedia.org/wiki/ToolLabsDatabasePlan https://wiki.toolserver.org/view/Database_access#Program_access [toolserver docs have these program access examples. would be nice to do something similar for Tool Labs]

8   Submitting, managing and scheduling Tool jobs on the grid 8.1 What is the grid? 8.2   Submitting jobs 8.2.1   Job names and memory allocation 8.2.2   Simple, one-off jobs 8.2.3   Continuous jobs (such as bots) 8.2.4  Output files 8.3   Managing jobs 8.3.1   Job status 8.3.2   Stopping jobs 8.3.3   Suspending and unsuspending jobs 8.4   Scheduling jobs at regular intervals with cron 8.4.1 Creating a chrontab 8.4.2   Specifying time zones >Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#The_grid_engine http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Job-system http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#2.29_Computing_grid http://lists.wikimedia.org/pipermail/labs-l/2013-July/001416.html https://wikitech.wikimedia.org/wiki/Help:Cron https://en.wikipedia.org/wiki/Cron

9. Developing on Tool Labs 9.1 Licensing 9.2 Heavy processing 9.3 Installing additional packages 9.4 Installing and running your bot 9.5 Where to put shared Tool code 9.6 Web services 9.6.1 Running scripts 9.6.2 Using cookies 9.6.3 Web logs 9.6.4 Tips and tricks (add them here!) 9.7 PyWikipediaBot 9.7.1 Setting up PyWikipediaBot for your account 9.7.2 Using PyWikipediaBot with virtualenv 9.7.3 Tips and tricks (add them here!) ... what else?... 9.8 Tips for working collaboratively .... do you have any? ... (1) Use source control

9.9 how to use  to write tools on labs' > Sources: http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Licenses:_Info https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Do_I_explicitly_have_to_specify_the_license_of_my_tools.3F (licencing) http://www.mediawiki.org/w/index.php?title=File:Tool_Labs_presentation_(Hackaton_2013).pdf&page=11 (heavy processing info) http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#1.29_Interactive_.2F_Development_environment (more on heavy processing) https://wikitech.wikimedia.org/wiki/Obsolete:Bots_project_documentation http://lists.wikimedia.org/pipermail/labs-l/2013-February/000903.html (shared tool code) https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Cookies https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Published_directories (running scripts) https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Log_your_actions http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Logs.2FStats https://wikitech.wikimedia.org/wiki/User:Russell_Blau/Using_pywikibot_on_Labs (pywikibot set up info) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001253.html (collaboratively working tip) https://wikitech.wikimedia.org/wiki/User:Legoktm/pywikibot_on_tools_lab (pywikipediabot in virutalenv) Also, if you ever want to submit a job via python, I wrote another simple library to help do that: https://wikitech.wikimedia.org/wiki/User:Legoktm/wmflib (from legoktm)

10   ·       Redis 10.1 Security 10.2 A note about memcache > Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Redis http://lists.wikimedia.org/pipermail/labs-l/2013-June/001354.html http://lists.wikimedia.org/pipermail/labs-l/2013-July/001413.html (memcache thread)

11. Public dataset dumps >Sources:https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Dumps https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Public_datasets https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#More

12. Monitoring your Tool 12.1  Web logs (access and error) 12.2 Web page statistics 12.3 Icinga system (status) 12.4 Ganglia system (status, workload) 12.5 Nagios system

> Sources: https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Log_your_actions http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Logs.2FStats http://nagios.wmflabs.org/nagios3/ http://ganglia.wikimedia.org/latest/ http://ganglia.wmflabs.org/latest/?c=tools http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=tools&style=detail https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en (info re web page stats) https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Logs.2FStats (re: apache and error logs)

13. Troubleshooting >Any tips? Please add them here! If you run into problems, please feel free to come into the #wikimedia-labs IRC (chat) channel using http://webchat.freenode.net/?channels=#wikimedia-labs and look for Coren (Marc-Andre Pelletier) or petan (Petr Bena). The labs-l mailing list at https://lists.wikimedia.org/mailman/listinfo/labs-l is another good place to ask for help, especially if the people in chat are not responding. You can also search wikitech.wikimedia.org for help pages, or look more widely with the custom search at https://www.google.com/cse/home?cx=010768530259486146519:twowe4zclqy.

14   ·      Backups 14.1 What gets backed up? The basic rule is: there is a lot of redundancy, but no backups of labs projects beyond the filesystem's time travel feature for short term disaster recovery. Labs users should make certain that they use source control to preserve their code, and make regular backups of irreplacable data. 14.2 Time travel >Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Backup https://blog.wikimedia.org/2013/05/30/preparing-for-the-migration-from-the-wikimedia-toolserver-to-tool-labs/ (notes on time travel) http://lists.wikimedia.org/pipermail/labs-l/2013-July/001426.html (instructions) 15   Moving a tool from Toolserver to Tool Labs (brief overview here, with a link to  "Migration of Toolserver tools" https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools

16   ·     FAQ  (additions to current list) Is there a way on Labs to easily share data across my Tool accounts? http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Filesystem_.2F_shared_storage Is there a GUI tool for database work? Not in Tool Labs, but you can run one locally on your computer (for example the MySQL Workbench http://dev.mysql.com/downloads/tools/workbench/). Here is how you connect to the database: For the login: username@tools-login.wmflabs.org For the database, it depends on the exact one you want to use, of course - for example: enwiki.labs https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools/Help#GUI_tool_for_databasework Why does public_html not work in my home directory? Users do not and cannot have a public_html folder to themselves. The only web accessible directories are in /data/project/ /public_html/*. To have a url such as http://tools.wmflabs.org/ / there must be a tool called and therefor a folder called /data/project/ /public_html/ https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ I get a Permission denied error when running my script. Why's that? Make sure that you are running your script from your tool account rather than your user account. https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ How can I detect if I'm running in Labs? How can I detect in which project (tools or toolsbeta) it's running? There is a file that contains the project name on every labs instance: /etc/wmflabs-project Testing for its presence tells you "you are in the WMF Labs", and checking its contents will tell you which project; that would be "tools" for the Tool Labs, or "toolsbeta" for the experimental tool labs. > http://lists.wikimedia.org/pipermail/labs-l/2013-July/001394.html My connection seems slow. Any advice? When connecting to Tool Labs from Europe you might have higher ping times. Try using mosh (mosh.mit.edu). To connect, use mosh -a tools-login.wmflabs.org (-a to force predictive echo) (instead of ssh tools-login.wmflabs.org) > https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Slow_connections I want to ssh to bot instances outside of Labs. Any advice? If you want to ssh to specific bot instances other than tools-login.wmflabs.org, it is helpful to create a new SSH key: $ ssh-keygen $ cat ~/.ssh/id_rsa.pub ssh-rsa .... user@host Copy the ssh-rsa ... user@host line to your authorized keys in the labs console (https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack ). > https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Connecting_to_different_servers

How can I check my filesystem usage? >https://wikitech.wikimedia.org/wiki/Help:Shared_storage <- Not answered there, says come ask Ryan on IRC. :)

Where can I see my page stats? > Webserver page view/visit statistics: https://graphite.wikimedia.org/ (requires Labs account), https://gdash.wikimedia.org/ (open to the world)

What languages are supported? > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Languages <- I think here the question has not been answered - i18n support for what exactly? This probably refers to the i18n support provided for by https://wiki.toolserver.org/view/Toolserver_Intuition which is currently being ported. (Krenair might have a status update)

Can I use any free license? You can use any OSI-approved license. Read more on the licenses on the Open Source Initiative's website: http://opensource.org/licenses > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Licenses:_Info

My Tool requires a package that is not currently installed in Tool Labs. How can I add it? You might not be the only one missing that package. Please submit a ticket in bugzilla and ask the admins to install it project-wide. If they have reasons not to do so you can always install software locally / just for yourself. > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bots_and_webservices_project

How can I change the permissions of a file in my Tool account? You cannot use chown. http://lists.wikimedia.org/pipermail/labs-l/2013-July/001377.html General documentation about your tool account is here: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Your_tool_account

Can I override the default VIM settings? > http://lists.wikimedia.org/pipermail/labs-l/2013-June/001294.html >http://lists.wikimedia.org/pipermail/labs-l/2013-June/001297.html

My tool needs a more specialized infrastructure than Tool Labs provides. What should I do? Tool Labs is a simplified environment intended to be a direct toolserver replacement for most small tools. When you need something more complicated or when you need to manage specialized infrastructure, what Tool Labs can't do you can probably do with your own Labs project! (Instead of inside one of the projects "tools" or "toolsbeta".) > http://lists.wikimedia.org/pipermail/labs-l/2013-June/001350.html

I keep reading about puppet. What is it? As a tool maintainer you don't have to worry about puppet.

The nutshell: puppet is a system by which you describe the configuration of a machine. When used, it will apply the necessary changes to make the machine you apply it to match that configuration.

In practice, the sysadmin would make any change of configuration intended for the machine in puppet (including what to install, files to edit, etc) so that it can be reapplied to a blank machine to configure it "just like it was", or to make a clone, and so on. In the case of a project where the tool maintainer do the system administration, it might be desirable to acually configure and install the tool /itself/ through puppet so that it is easy to return to a known state.

In the case of the Tool Labs, however, the actual tools would not normally be configured through puppet (it's possible, but not worthwile): they live on a shared filesystem rather than on the individual machines. What puppet /is/ used for is to maintain the components of the grid, making adding "one more compute node" or "an extra webserver" as simple as to create a new instance and set puppet accordingly. When we find a tool has a dependency, the tools labs sysadmins will add it to puppet so that every host part of the grid (current and future) will have that configured accordingly without manual intervention. (source: http://lists.wikimedia.org/pipermail/labs-l/2013-March/001047.html)

What is the labsconsole? [is it important to keep this concept? I think it's worth the sentence below because in some places the docs still refer to "labs console".] https://wikitech.wikimedia.org/wiki/Help:Access It's the old name of what is now Wikitech (wikitech.wikimedia.org)

I'm being prompted for a password when I try to 'become my-tool-account'. What's wrong? If you are seeing "a password is required" message when you try to become your tool (i.e., sudo yourtoolaccount), it is likely because you were logged in to your Labs account when you created the tool account. Unix group membership is checked on login only, so an existing session will not have access to the new tool group. Log out and then log in to your Labs account again to fix this problem.



Moving from Toolserver to Tool Labs: https://www.mediawiki.org/wiki/Wikimedia_Labs/Migration_Of_Toolserver_Tools This is a very helpful doc.

...I suggest we collect more toolserver-transition specific help in this document and link to it from Tool Labs Guide...

Modifications: Replace "bot project" with "toolsbeta project" (the current name of the project)

(modify) 6.9 What's a "service group"? --> Note that Tool Labs terminology is "Tools account" Perhaps clarify: "service group" is a general Labs concept; "tool account" is what we use service groups for in the Tool Labs project.

Additional help topics: 6.18 What should I do about Toolserver links? > http://lists.wikimedia.org/pipermail/toolserver-l/2013-May/006143.html (.htacess redirect info)

6.19 Anything I should know about Ubuntu Linux? Not familiar with Ubuntu Linux? Below, you can find some good sources of information. You have to do with a Debian-based Linux distribution. Basically if there are different documents you'll probably want to look for the server (not the desktop) edition. The current stable release that has long term support is "Ubuntu Precise", a.k.a "Precise Pangolin", a.k.a. "12.04". This is what's running in Tool Labs. Once you know what a software/command line tool is called you can read the manual directly in Tool Labs by entering "man ".

Official Ubuntu Documentation: https://help.ubuntu.com/ Find groups or IRC channels in your language: http://loco.ubuntu.com/teams/

Stuff in German: http://wiki.ubuntuusers.de - very good explanations of software in German, command line as well as graphical tools. (In contrast, the forum.ubuntuusers.de contains relatively much bullshit.)

Please add more links to docs in different languages if you can really recommend them! > Context for question: http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#End-user-support > Source for answer: You! 6.20 Can I move my SVN repository? > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Version_control > http://lists.wikimedia.org/pipermail/toolserver-l/2013-June/006175.html > http://lists.wikimedia.org/pipermail/toolserver-l/2013-June/006177.html

6.22 Will Tool Labs have meta database tables like the ones in Toolserver? > See thread: http://lists.wikimedia.org/pipermail/labs-l/2013-July/001390.html https://bugzilla.wikimedia.org/show_bug.cgi?id=48626 https://bugzilla.wikimedia.org/show_bug.cgi?id=48625

6.23 Can I join user databases with wiki ones? https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features That's actually working today; databases created by users can be joined with the replica databases so long as they are created on the same DB (i.e.: connect to the same place)

6.24 Will the commons database be replicated to all clusters, like it is on the Toolserver? https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features It is not. However, there is a federated database link to commons and wikidata on every shard. There are some caveats in using them for good performance, but they are available. (Needs documenting)

6.25 Can I import my JIRA bug tracker information to Bugzilla? If so, how? No, sorry. At the moment, we don't have a solution for you to keep the bugreport from jira. Some people have done such a migration but this would have to be thoroughly tested and would probably need a lot of manual intervention. The good news is: If you wish you can request to have your own "component" in bugzilla for supporting your tool. > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bug_tracker > any other suggestions?

6.26 Does the Tool Labs grid support qcronsub? >https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#qsub_et_al Not directly. Similar functionality is available with the -once option to jsub but the common use case of "continually try to restart my tool if it failed" is better served with the support for continuous applications (-continuous, jstart)

Can I move a multimaintainer project to Tool Labs? >https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuff The short answer; all Tool Labs tools are "multimaintainer" as meant on toolserver (i.e.: shared project space, group membership)

My bot has many maintainers. How should we handle credentials? > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuff Credentials the bot itself uses to edit/perform actions is subject to the target wiki's policies on sharing credentials and may need to be cleared with them.

Does Tool Labs support all the packages Toolserver does? >https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bots_and_webservices_project (Is this yes? or were requested packages dif. from total ones?) > https://wiki.toolserver.org/view/User:Dab/Debian-Packages > https://jira.toolserver.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+TS+AND+%28summary+~+install+OR+description+~+install+OR+comment+~+install%29+AND+issuetype+%3D+Task+AND+resolution+%3D+Fixed It's a "yes" insofar as "If you need it and it is open source, we can get it for you." We have not, on the other hand, preemptively every version of every dependency that has been enumerated; but do so on demand (partly to avoid installing obsolete packages that are no longer relevant).



INVENTORY OF CURRENT TOOL-LAB RELATED DOCUMENTATION, AND PROPOSED UPDATE/REDIRECT PLAN:

https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [UPDATE/REPLACE WITH NEW, EXPANDED GUIDE OUTLINED ABOVE]

https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools [UPDATE WITH HIGH-LEVEL OVERVIEW/INFO ABOUT CREATING ACCOUNT] Sources for expansion: :http://www.mediawiki.org/wiki/File:Tool_Labs_presentation_(Hackaton_2013).pdf

https://wikitech.wikimedia.org/wiki/Main_Page [UPDATE: Under Labs/add note about Tool Labs and add link to Tools project and Tool Labs guide]

https://wikitech.wikimedia.org/wiki/Help:Terminology [UPDATE with additional terms--Tool Labs, Toolsbeta project, Tools project, grid, cluster... other terms?]

http://tools.wmflabs.org/ [UPDATE to clarify what these links are (e.g. "Request access to Tool Labs"; add additional context at top of page to explain what is on page]

https://www.mediawiki.org/wiki/Wikimedia_Labs [Appears current/recently updated]

https://www.mediawiki.org/wiki/Developer_access [UPDATE with context-- what does 'developer access' mean exactly? ]

https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features [update]

https://wikitech.wikimedia.org/wiki/Help:Getting_Started [Appears current]

https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta [UPDATE with more info about Toolsbeta and how to get access]

https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Project_storage [UPDATE: there's a TODO in the page]

https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools [Update: replace 'bot' with 'toolsbeta'/also add content generated/organized in sprint]

https://www.mediawiki.org/wiki/Wikimedia_Labs/TODO [Appears current]

http://www.mediawiki.org/wiki/Toolserver/List_of_Tools [Appears current] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design [Appears current]

https://wikitech.wikimedia.org/wiki/Help:Shared_storage [appears current]

https://www.mediawiki.org/wiki/Special:PrefixIndex/Wikimedia_Labs/: https://www.mediawiki.org/wiki/Wikimedia_Labs/Authentication_improvement_project [Appears current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Instance_creation_improvement_project [Appears relatively current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ [recently updated, but would love to redirect this to guide/combine info there] https://www.mediawiki.org/wiki/Wikimedia_Labs/Create_shared_sql_service_for_all_projects (last updated jan 2012) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Per-project_saltstack_remote_execution (last update june 2012) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Shared_home_directories_per_project (last update dec 2011) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/TODO (Appears current) https://www.mediawiki.org/wiki/Wikimedia_Labs/Things_to_fix_in_beta (Last updated June 2012) Redirect to roadmap/ https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/List_of_Toolserver_Tools/tool [Appears current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Log_bot_for_wikimedia-labs_channel (last update Dec 2011) Redirect https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Nagios_management_without_exported_puppet_resources (last update Dec 2011) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en [Update/to replace 'bots' with 'toolsbeta' to avoid confusion] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Rules [Update with rules] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs [Update overview at top of page] https://www.mediawiki.org/wiki/Wikimedia_Labs/Reverse_proxy_for_web_services [last update march 2012/redirect to roadmap] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design (appears current) https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/List_of_Toolserver_Tools [appears current]