User:Kmenger/ToolLabsSprint

Plan: July 23-25, a bunch of us are going to work on documentation, here on Etherpad, on the wikitech.wikimedia.org wiki, and in #wikimedia-labs on Freenode. Join us   :)Participants include:KirstenSilke (part of the time)SumanaRyanCorenJohannes (don't have lots of time, but would like to add some documentation about catgraph)Addshore (don't have lots of time, but will try to keep track of what happening and participate when and where i can) [=(add yourself)This document contains:1. An initial stab at an outline for a Tool Labs guide, along with source links--to draw from or point to--for each section2. Additions to the "Migrating of Toolserver guide", with source links3. An inventory of current documentation relating to Tool Labs, along with a proposal for a) updating b) redirection c) Appears current All help much appreciated! I also have some questions/notes here:  http://www.mediawiki.org/wiki/User:Kmenger/Tool_Labs*******************************************************************************************************Tool Labs guide1    What is Tool Labs Tool Labs is a reliable, scalable hosting environment for community developers working on tools and bots that help users maintain and use wikis. The cloud-based infrastructure was developed by the Wikimedia Foundation and is supported by a dedicated group of Wikimedia Foundation staff and volunteers. Tool Labs is a part of the Labs project, which is designed to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure, including MediaWiki, and to do analytics and bot work.1.1    RationaleTool Labs was developed in response to the need to support external tools and their developers and maintainers. The system is designed to make it easy for maintainers to share responsibility for their tools and bots, which helps ensure that no useful tool gets  ‘orphaned’ when one person needs a break. The system is designed to be reliable, scalable and simple to use, so that developers can hit the ground and start coding. 1.2    FeaturesIn addition to providing a well supported hosting environment, Tool Labs provides:·       support for Web services, continuous bots, and scheduled tasks·       access to replicated production databases·       easily shared management of tool accounts, where tools and bots are stored·       a grid engine for dispatching jobs·       support for mosh, SSH, SFTP without complicated proxy setup ·       time-travel backups for short-term data recovery·       Ganglia, Icinga, and Nagios systems to systems to help monitor tools ·       Version control via Gerrit and Git·       1.3    Architecture and terminologyTool Labs has essentially four components: the bastion hosts, the grid, the web cluster, and the databases. Users access the system via one of two Tool Lab projects: ‘tools’ or ‘toolsbeta’. To request an account on the ‘tools’ project, where most tool and bot development is hosted and maintained, please see Special:FormEdit/Tools Access Request 1.3.1 Bastion hosts, grid, web cluster, databases Bastion hostsThe bastion host is where users log in to Tool Labs. Currently, Tool Labs has two bastion hosts: tools-login.wmflabs.org  tools-dev.wmflabs.orgThe two hosts are functionally identical, but we request that heavy processing (compiles, etc) be done only on tools-dev.wmflabs.org to keep interactive performance on tools-login.wmflabs.org snappy. The gridThe Tool Labs grid, implemented with Open Grid Engine (the open-source fork of Sun Grid Engine) permits users to submit jobs from either a log-in account on the bastion host or from a Web service. Submitted jobs are added to a work queue, and the system finds a host to execute them. Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once. If a continuous job fails, the grid will automatically restart the job so that it keeps going.For more information about the grid, please see [#Submitting, managing and scheduling Tool jobs on the grid]The Web clusterThe Tool Labs Web cluster is fronted by a Web proxy, which supports SSL and is open to the Internet. Any of the servers in the cluster can serve any of the hosted Web tools as Tool Labs uses a shared storage system; the proxy distributes between the Web servers. The cluster uses suPHP (www.suphp.org/) to run scripts and CGI, and will soon support WSGI (www.wsgi.org/‎). Note that individual tool accounts have both a  ~/public_html/  and a ~/cgi-bin/ directory in the home directory for storing Web files. For more information, please see [#Web services] The databases Tool Labs supports two sets of databases: the production replicas and user-created databases, which are used by individual tools. The production replicas follow the same setup as production, and the information that can be accessed from them is the same as that which normal registered users (i.e.: not  +sysop or other types of advanced permissions) can access on-wiki or via the API. Note that some data has been removed from the replicas for privacy reasons.User-created databases can be created by either a user or a tool on the replica servers or on a local ‘tools’ project database. 1.3.2 Projects: Tools and ToolsbetaLike the rest of Labs, Tool Labs is organized into ‘projects’. Currently, Tool Labs consists of two projects: ‘tools’ and ‘toolsbeta’, which are described in more detail here: Tools  project:  https://wikitech.wikimedia.org/wiki/Nova_Resource:ToolsToolsbeta  project: https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta  The ‘tools’ project is where tools and bots are developed and maintained. ‘Toolsbeta’ is used for experiments in the Tool Labs environment itself--things like new systems or experimental versions of system libraries that could affect other users. In general, every tool maintainer should work primarily on the "tools" project, only doing work on toolsbeta when changes to Tool Labs itself need to be tested to support their tool.1.3.3 InstancesDevelopers working in Tool Labs do not have to create or set up virtual machines (i.e., Lab ‘instances') as the Tool Lab project admins create and manage them. The term will come up in the Labs documentation; otherwise, don’t worry about it.1.4    Rules of use1.4.1 Tool Labs policiesAll tools and bots developed and maintained on Tool Labs must adhere to the terms of use that will be available here when they are finalized: Tool Labs > Rules ( https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Rules)Specifically, tools must be·       Open source·       Open dataPrivate information must be handled carefully, if at all. Note that private user information has been redacted from the replicated databases provided by the system. As the Tool Labs environment is shared, we ask that you strive not to break things for others, and to be considerate when using system resources.For more information about general Labs policies, please see Wikimedia Labs > Terms of use (  https://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use) [MAP: Those terms of use are for "generic" Labs users, the Tool Labs will have its own different TOU since the audience isn't the same]1.4.2 Individual wiki policies (these differ!)When developing on Tool Labs, please adhere to the bot policies of the wikis your bot interacts with. Each wiki has its own guidelines and procedures for obtaining approval. The English Wikipedia, for example, requires that a bot be approved by the Bot Approvals Group before it is deployed, and that the bot account be marked with a ‘bot’ flag. . For general information and guidelines, please see ‘Bot policy’  1.5    ContactWe’d love to hear from you! You can find us here:On IRC: #wikimedia-labs on Freenode A great place to ask questions, get help, and meet other Tool Lab developers. See Help:IRC for more information. https://wikitech.wikimedia.org/wiki/Help:IRCVia mailing list: Labs-l@lists.wikimedia.org A list for announcements and discussion related to the Wikimedia Labs project. You can find the archives here:  http://lists.wikimedia.org/pipermail/labs-l/  Found a bug?: Bugs can be posted to Bugzilla:  https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs  > Sources: :http://www.mediawiki.org/wiki/File:Tool_Labs_presentation_(Hackaton_2013).pdf                        https://wikitech.wikimedia.org/wiki/Help:Terminology                               http://meta.wikimedia.org/wiki/Bot                               http://en.wikipedia.org/wiki/Wikipedia:Bot_Approvals_Group                        https://meta.wikimedia.org/wiki/Bots                   https://meta.wikimedia.org/wiki/Bot_policy                   https://www.mediawiki.org/wiki/Wikimedia_Labs                  https://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use                 https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools                 https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta              http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005332.html             http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Languages             http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Projects  (tools/toolsbeta info)             https://wikitech.wikimedia.org/wiki/Help:IRC             https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs  (bugzilla)2    Getting access to Tool LabsAnyone can view the source code and the output of most tools and bots, and anyone can get an account of their own as well.To access Tool Labs you need:to create a Labs account, which provides shell access (you must upload an SSH key)to request access to the 'tools' project       Steps for creating a Labs account,  creating and uploading an SSH key, and for requesting access to the 'tools project' are described in the next sections.2.1    Creating a Labs account on WikitechBefore you can access Tool Labs, you must create a Labs account on Wikitech, which is the general interface for everything Labs.Sign up for a Labs account here:  https://wikitech.wikimedia.org/wiki/Special:UserLogin/signupThe "Instance shell account name" you specify in the Create Account form will be your Unix username on all Labs projects. If you forget your username, you can always find it under Preferences > Instance shell account name. Once you have created a Labs account you will be added to a list of users to be approved for shell access, which you can see here: Shell Access Requests. ( https://wikitech.wikimedia.org/wiki/Category:Shell_Access_Requests)2.2     Generating and uploaing an SSH keyIn order to access Labs servers using SSH, you must provide a public SSH key. Once you have created a Labs account, you can specify a public key on the 'OpenStack' tab of your Wikitech preferences.Specify the SSH key here:  https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack2.2.1     Generating a key in WindowsTo generate an SSH key in Windows:1. Open PuttyGen2. Select an SSH-2 RSA key3. Click the Generate button4. Move your mouse around until the progress bar is full5. Type in a passphrase (you will need to remember this) and confirm it6. Save the private key and public key onto your local machine7. From the text field 'Public key for pasting into OpenSSH authorized_keys file' right click and copy8. Inset this into your 'OpenStack' tab of your Wikitech preferences2.2.2    Generating a key in LinuxModern Unix systems include the OpenSSH client (if not then install it). To generate a key, use: ssh-keygen -t rsaThis will store your private key in $HOME/.ssh/id_rsa, and your public key in $HOME/.ssh/id_rsa.pub. You can use different filenames, but these are the default filenames, so it's easiest to not change them.2.3    Requestng access to the 'tools' projectOnce you have created a Labs account, you must request access to the ‘tools’ project by submitting a Tools Access Request.Submit  a Tools Access Request here:  https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_RequestRequests for access are generally dealt with within the day (often faster), though response-time may be longer depending on admin availability. If you need immediate assistance, please contact us on IRC.2.4  Receiving access to the 'tools' projectOnce your 'tools' project access request has been processed, you will become a member of the 'tools' project, and will be able to  access it using the "Instance shell account name" provided when creating your Labs account and the private key matching the public key you supplied for authentication. For more information about accessing the project, please see #Accessing Tool Labs.2.4.1 NotificationYou will be notified on Wikitech that your user rights were changed, that your request was linked from 'Nova Resource:Tools', and that you have been added to the project Nova Resource:Tools. You will also receive email explaining that your user rights have been changed, and that you are now a member of the group 'shell'. In other words, your Tool Labs account is ready for you to use!2.4.2 Storage and useAlthough you  access Tool Labs via your Labs account,  we strongly recommend against saving data or tools in any space  that is accessible to individuals only. Tools and bots should be maintained in Tool accounts, which have flexible memberships (i.e., multiple people can help maintain the code!). For more information about Tool accounts, please see [#Joining and creating a Tool account]>Sources:  https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Getting_access                     https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_instances  (advice about SSH keys)                    https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents  (advice about multiple SSH keys)                     https://www.mediawiki.org/wiki/Wikimedia_Labs/Account_creation_text                      https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Create_a_User_Account                     https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Home_directory_storage                     http://lists.wikimedia.org/pipermail/toolserver-l/2013-April/005952.html  (re 2.4.1: note to clarify notification/initial experience)                   https://wikitech.wikimedia.org/wiki/Help:Shared_storage  (not using indiv. account for shared storage)3    Accessing Tool Labs Tool Labs can be accessed in a variety of ways--from its public IP to a GUI client. Please see Help:Access ( https://wikitech.wikimedia.org/wiki/Help:Access  ) for general information about accessing Labs. Pointers to more information on specific means of access below.3.1    Tools home page  The Tools home page: http://tools.wmflabs.org/The Tools home page is publicly available and contains a list of all currently hosted Tool accounts along with the name(s) of the maintainers for each. Individual tool accounts that have an associated web page will appear as links. Users with access to the 'tools' project can create new tool accounts here, and add or remove maintainers to and from existing tool accounts.3.2    SSH/SFTP/SCP Users can SSH to the 'tools' project via its bastion host: tools-login.wmflabs.org, provided that  a public SSH key has been uploaded to the Labs account. ssh your-instance-shell account name@tools-login.wmflabs.orgNote that if you plan to do heavy processing (compiling, etc), you should SSH to tools-dev.wmflabs.org. 3.2.1 Using 'take' to transfer ownership of uploaded filesOnce you have logged in via SSH, you can transfer files via sftp and scp. Note that the transferred files will be owned by you. You will likely wish to transfer ownershihp to your tool account. To do this:1. Become your tool account using 'become':maintainer@tools-login:~$ become toolaccount local-toolaccount@tools-login:~$2. As your tool account,  'take' ownership of the files:local-toolaccount@tools-login:~$ take FILEThe 'take' command will change the ownership of the file(s) and directories recursively to the calling user (in this case, the tool account).3.2.2 Using multiple ssh agentsIf you use multiple ssh-agents (to connect to your personal or company system, for example), see Managing Multiple SSH Agents ( https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents  ) for more information about setting up a primary and a Labs agent.3.4    Putty and WinSCPNote that instructions for accessing Tool Labs with Putty and WinSCP differ from the instructions for using them with other Labs projects. Please see Help:Access to ToolLabs instances with PuTTY and WinSCP ( https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP)  for information specific to Tool Labs.3.5    Other graphical file managers (e.g., Gnome/KDE)For information about using a graphical file manager (e.g., Gnome/KDE), please see Accessing Tool Labs > Accessing instances with a graphical file manager ( https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_a_graphical_file_manager)>Sources: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_instances                     https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCPhttps://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_a_graphical_file_manager http://www.mail-archive.com/labs-l@lists.wikimedia.org/msg00375.html  for Gnome/KDE users and https://wikitech.wikimedia.org/wiki/User:Wikinaut/Help:Access_to_instances_with_PuTTY_and_WinSCP  for Windows users https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Access_Project_Instanceshttps://wikitech.wikimedia.org/wiki/Help:Accesshttps://wikitech.wikimedia.org/wiki/Help:Puttyhttps://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#access_to_instances:_http.2Fsftphttps://wiki.toolserver.org/view/Transferring_files  (toolserver docs on transferring files)4    Joining and creating a Tool account4.1 What is a Tool account?Tool accounts, which can be created by any ‘tools’ project member, are fundamental to the structure and organization of Tool Labs. Although each tool account has a user ID, they are not personal accounts (like a Labs account), rather services that consist of a user and group ID (i.e., a unix uid-gid pair) that are intended to run the actual tool or bot. > Unix user: local-toolname> Unix group: local-toolnameMembers of the Unix group include:>  the tool account creator >  the tool account itself>  (optionally, but encouraged!) additional tool maintainers Maintainers may have more than one tool account, and tool accounts may have more than one maintainer. Every member of the group has the authorization to sudo to the tool account. A simple way for maintainers to switch to the tool account is with ‘become’:maintainer@tools-login:~$ become toolname  local-toolname@tools-login:~$ In addition to the user/group pair, each tool account includes:·       A home directory on shared storage: /data/project/toolname·       A ~/public_html/ and ~/cgi-bin/ directory, which are visible at   http://tools.wmflabs.org/toolname/  and http://tools.wmflabs.org/toolname/cgi-bin, respectively·       Database access credentials: replica.my.cnf, which provide access to the production database replicas as well as to project-local databases. ·       Access to the continuous and task queues of the compute grid 4.2  Joining an existing Tool accountAll tool accounts hosted in Tool Labs are listed on the Tools home page ( http://tools.wmflabs.org/). If you would like to be added to an existing account, you must contact the maintainer(s) directly.If you would like to add (or remove) maintainers to a tool account that you manage, you may do so with the 'add' link found beneath the tool name on the Tools home page.4.3  Creating a new Tool accountMembers of the ‘tools’ project can create tool accounts from the Tools home page:1. Navigate to the Tools home page:  http://tools.wmflabs.org/2. Select the “create new tool” link (found beside “Hosted tools” near the top of the page3.     Enter a  “Service group name”. The service group name will be used as the name of your tool account. Do not prefix your service group name with local-; the management interface will do so automatically where appropriate, and there is a known issue that will cause the account to be created improperly if you do.Note: If you have only recently been added to the ‘tools’ project, you may get an error about not having appropriate credentials. Simply log out and back in to Wikitech to fix thisThe tool account will be created and you will be granted access to it within a minute or two. If you were already logged in to your Labs account through SSH, you will have to log off then back in before you can access the tool account.      4.4 Deleting a Tool account       You can't delete a tool account yourself, though you  can delete the content of your directories. If you really want a tool account to be deleted, please contact an admin. 4.5 Using Toolsbeta     Nearly all tool development is done on the 'tools' project, and 99.9% of the time, creating a tool account on this project will serve your  needs. However, if your tool or bot requires an experimental library or a significant change to the 'tools' infrastructure--anything that  could potentially negatively impact existing tools--you should experiment with the new infrastructure on toolsbeta. To request access to toolsbeta, please visit #wikimedia-labs on IRC. You can also request acces via the labs-l mailing list or via bugzilla.>Sources:  https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Your_tool_account                    https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Create_a_new_tool                     https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Project_storage                     https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Join_Projects                     https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools  [note time lag btw tool account creation and user becomming tool] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Published_directorieshttps://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Can_I_have_a_subdomain_for_my_web_service.3Fhttps://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_accesshttp://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#Other_implementation_details  [note about requesting tool resources] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools#Can_I_delete_a_tool.3F                          https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuff  (on bot credentials/multiple maintainer tool accounts)                  http://lists.wikimedia.org/pipermail/labs-l/2013-June/001311.html  (info on toolsbeta 'bots')                  http://lists.wikimedia.org/pipermail/labs-l/2013-June/001313.html  (more on toolsbeta)                 http://lists.wikimedia.org/pipermail/labs-l/2013-May/001194.html  (more on toolsbeta)                 http://lists.wikimedia.org/pipermail/labs-l/2013-May/001198.html  (more on toolsbeta)                https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta                  http://lists.wikimedia.org/pipermail/labs-l/2013-March/001018.html  (re: new package install)5    Customizing a Tool account5.1    Creating a tool description and web page5.2    Configuring mail5.3    A few notes about permissions... any other things people would like to use/configure?...> Sources:   https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Tool_Labs.27s_landing_page                       https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Mail                       http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Mail:_to_do                       https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Tool_accounts                       https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools#What_about_file_permissions.3F_Who_can_see_my_code.3F                      http://lists.wikimedia.org/pipermail/labs-l/2013-May/001177.html & http://lists.wikimedia.org/pipermail/labs-l/2013-May/001179.html  (note about permission)                     http://lists.wikimedia.org/pipermail/labs-l/2013-May/001181.html  (note about permission for public_html)                      http://lists.wikimedia.org/pipermail/labs-l/2013-February/000886.html  (idea for what to put in html page)                          6. Setting up code review and version controlAlthough it's possible to just stick your code in the directory and mess with it manually every time you want to change something, your future self and your future collaborators will thank you if you instead use source control, a.k.a. version control and a code review tool. Wikimedia Labs makes it pretty easy to use Git for source control and Gerrit for code review, but you also have other options.6.1 Gerrit/Git 6.2. Other options> Sources:   https://www.mediawiki.org/wiki/Gerrit/New_repositories  (gerrit)                      https://www.mediawiki.org/wiki/Gerrit                       https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver  (other versioning options)                      https://wikitech.wikimedia.org/wiki/Help:Getting_Started#to_Wikimedia_Code_Review                      https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Git_committers7     Database access 7.1    Database replicas and naming conventions7.2    Connecting to the replicated databases7.3    Creating databases7.3.1    Steps to create a user database on tools-db7.3    Joins between commons and wikidata and other project databases> Sources:  https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access                      http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features  (status of join)                     http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005382.html                       http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Database_replication.2Faccess                       https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Create_new_databases                      https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Replicated_databases                      https://wikitech.wikimedia.org/wiki/ToolLabsDatabasePlan                      https://wiki.toolserver.org/view/Database_access#Program_access  [toolserver docs have these program access examples. would be nice to do something similar for Tool Labs]                8    Submitting, managing and scheduling Tool jobs on the grid8.1 What is the grid?8.2    Submitting jobs8.2.1    Job names and memory allocation8.2.2    Simple, one-off jobs8.2.3    Continuous jobs (such as bots)8.2.4   Output files8.3    Managing jobs8.3.1    Job status8.3.2    Stopping jobs8.3.3    Suspending and unsuspending jobs8.4    Scheduling jobs at regular intervals with cron8.4.1  Creating a chrontab8.4.2    Specifying time zones>Sources:  https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#The_grid_engine                      http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Job-system                     http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#2.29_Computing_grid                     http://lists.wikimedia.org/pipermail/labs-l/2013-July/001416.html                     https://wikitech.wikimedia.org/wiki/Help:Cron                      https://en.wikipedia.org/wiki/Cron9. Developing on Tool Labs9.1 Licensing9.2 Heavy processing 9.3  Installing additional packages 9.4 Installing and running your bot9.5  Where to put shared Tool code9.6 Web services9.6.1 Running scripts9.6.2  Using cookies9.6.3  Web logs9.6.4 Tips and tricks (add them here!)9.7 PyWikipediaBot9.7.1 Setting up PyWikipediaBot for your account9.7.2 Using PyWikipediaBot with virtualenv9.7.3 Tips and tricks (add them here!)... what else?...9.8 Tips for working collaboratively       .... do you have any? ...(1) Use source control9.9  how to use  to write tools on labs'> Sources:  http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Licenses:_Infohttps://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Do_I_explicitly_have_to_specify_the_license_of_my_tools.3F  (licencing)    http://www.mediawiki.org/w/index.php?title=File:Tool_Labs_presentation_(Hackaton_2013).pdf&page=11  (heavy processing info)                     http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#1.29_Interactive_.2F_Development_environment  (more on heavy processing)                     https://wikitech.wikimedia.org/wiki/Obsolete:Bots_project_documentation http://lists.wikimedia.org/pipermail/labs-l/2013-February/000903.html  (shared tool code)                       https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Cookies                        https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Published_directories  (running scripts) https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Log_your_actions                      http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Logs.2FStatshttps://wikitech.wikimedia.org/wiki/User:Russell_Blau/Using_pywikibot_on_Labs  (pywikibot set up info) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001253.html  (collaboratively working tip) https://wikitech.wikimedia.org/wiki/User:Legoktm/pywikibot_on_tools_lab  (pywikipediabot in virutalenv)Also, if you ever want to submit a job via python, I wrote another simple library to help do that:  https://wikitech.wikimedia.org/wiki/User:Legoktm/wmflib  (from legoktm)10    ·       Redis10.1 Security10.2  A note about memcache> Sources:  https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Redis                      http://lists.wikimedia.org/pipermail/labs-l/2013-June/001354.html                       http://lists.wikimedia.org/pipermail/labs-l/2013-July/001413.html  (memcache thread)11. Public dataset dumps>Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Dumps                    https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Public_datasets                    https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#More12. Monitoring your Tool12.1   Web logs (access and error)12.2  Web page statistics 12.3  Icinga system (status)12.4  Ganglia system (status, workload)12.5 Nagios system> Sources:  https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Log_your_actions                      http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Logs.2FStats                       http://nagios.wmflabs.org/nagios3/                         http://ganglia.wikimedia.org/latest/                        http://ganglia.wmflabs.org/latest/?c=tools                       http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=tools&style=detail                      https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en  (info re web page stats)                    https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Logs.2FStats  (re: apache and error logs)>> Can anyone point to more info or briefly state when/and how best to use nagios, ganglia/icinga (i.e., one instead of the other; what the benefits are, etc)?<<13. Troubleshooting>Any tips? Please add them here!If you run into problems, please feel free to come into the #wikimedia-labs IRC (chat) channel using  http://webchat.freenode.net/?channels=#wikimedia-labs  and look for Coren (Marc-Andre Pelletier) or petan (Petr Bena). The labs-l mailing list at  https://lists.wikimedia.org/mailman/listinfo/labs-l  is another good place to ask for help, especially if the people in chat are not responding. You can also search wikitech.wikimedia.org for help pages, or look more widely with the custom search at  https://www.google.com/cse/home?cx=010768530259486146519:twowe4zclqy  .14    ·      Backups14.1 What gets backed up?The basic rule is: there is a lot of redundancy, but no backups of labs projects beyond the filesystem's time travel feature for short term disaster recovery.Labs users should make certain that they use source control to preserve their code, and make regular backups of irreplacable data.14.2 Time travel      >Sources:  https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools                            http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Backup                           https://blog.wikimedia.org/2013/05/30/preparing-for-the-migration-from-the-wikimedia-toolserver-to-tool-labs/  (notes on time travel)                          http://lists.wikimedia.org/pipermail/labs-l/2013-July/001426.html  (instructions)      15    Moving a tool from Toolserver to Tool Labs (brief overview here, with a link to  "Migration of Toolserver tools"  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools16     ·     FAQ  (additions to current list) Is there a way on Labs to easily share data across my Tool accounts? http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Filesystem_.2F_shared_storage   Is there a GUI tool for database work?Not in Tool Labs, but you can run one locally on your computer (for example the MySQL Workbench  http://dev.mysql.com/downloads/tools/workbench/). Here is how you connect to the database:For the login: username@tools-login.wmflabs.orgFor the database, it depends on the exact one you want to use, of course - for example: enwiki.labs   https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools/Help#GUI_tool_for_databasework   Why does public_html not work in my home directory? Users do not and cannot have a public_html folder to themselves. The only web accessible directories are in /data/project/ /public_html/*.To have a url such as  http://tools.wmflabs.org/ / there must be a tool called and therefor a folder called /data/project/ /public_html/   https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ   I get a Permission denied error when running my script. Why's that?Make sure that you are running your script from your tool account rather than your user account. https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ     How can I detect if I'm running in Labs? How can I detect in which project (tools or toolsbeta) it's running?There is a file that contains the project name on every labs instance: /etc/wmflabs-projectTesting for its presence tells you "you are in the WMF Labs", and checking its contents will tell you whichproject; that would be "tools" for the Tool Labs, or "toolsbeta" for theexperimental tool labs.>  http://lists.wikimedia.org/pipermail/labs-l/2013-July/001394.html     My connection seems slow. Any advice?When connecting to Tool Labs from Europe you might have higher ping times. Try using mosh (mosh.mit.edu). To connect, usemosh -a tools-login.wmflabs.org (-a to force predictive echo)(instead of ssh tools-login.wmflabs.org)>  https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Slow_connections         I want to ssh to bot instances outside of Labs. Any advice?If you want to ssh to specific bot instances other than tools-login.wmflabs.org, it is helpful to create a new SSH key: $ ssh-keygen$ cat ~/.ssh/id_rsa.pubssh-rsa .... user@hostCopy the ssh-rsa ... user@host line to your authorized keys in the labs console ( https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack ).> https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Connecting_to_different_serversHow can I check my filesystem usage?> https://wikitech.wikimedia.org/wiki/Help:Shared_storage  <- Not answered there, says come ask Ryan on IRC. :)Where can I see my page stats?> Webserver page view/visit statistics:  https://graphite.wikimedia.org/  (requires Labs account),  https://gdash.wikimedia.org/  (open to the world)What languages are supported?>  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Languages  <- I think here the question has not been answered - i18n support for what exactly?This probably refers to the i18n support provided for by  https://wiki.toolserver.org/view/Toolserver_Intuition  which is currently being ported.  (Krenair might have a status update)Can I use any free license?You can use any OSI-approved license. Read more on the licenses on the Open Source Initiative's website:  http://opensource.org/licenses> https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Licenses:_InfoMy Tool requires a package that is not currently installed in Tool Labs. How can I add it?You might not be the only one missing that package. Please submit a ticket in bugzilla and ask the admins to install it project-wide. If they have reasons not to do so you can always install software locally / just for yourself.>  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bots_and_webservices_projectHow can I change the permissions of a file in my Tool account?You cannot use chown. http://lists.wikimedia.org/pipermail/labs-l/2013-July/001377.htmlGeneral documentation about your tool account is here:  https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Your_tool_accountCan I override the default VIM settings?>  http://lists.wikimedia.org/pipermail/labs-l/2013-June/001294.html>http://lists.wikimedia.org/pipermail/labs-l/2013-June/001297.htmlMy tool needs a more specialized infrastructure than Tool Labs provides. What should I do?Tool Labs is a simplified environment intended to be a direct toolserver replacement for most small tools. When you need something more complicated or when you need to manage specialized infrastructure, what Tool Labs can't do you can probably do with your own Labs project! (Instead of inside one of the projects "tools" or "toolsbeta".)>  http://lists.wikimedia.org/pipermail/labs-l/2013-June/001350.htmlI keep reading about puppet. What is it?As a tool maintainer you don't have to worry about puppet.The nutshell:  puppet is a system by which you describe the configuration of a machine. When used, it will apply the necessary changes to make the machine you apply it to match that configuration.In practice, the sysadmin would make any change of configuration intended for the machine in puppet (including what to install, files to edit, etc) so that it can be reapplied to a blank machine to configure it "just like it was", or to make a clone, and so on. In the case of a project where the tool maintainer do the system administration, it might be desirable to acually configure and install the tool /itself/ through puppet so that it is easy to return to a known state.In the case of the Tool Labs, however, the actual tools would not normally be configured through puppet (it's possible, but not worthwile):  they live on a shared filesystem rather than on the individual machines. What puppet /is/ used for is to maintain the components of the grid, making adding "one more compute node" or "an extra webserver" as simple as to create a new instance and set puppet accordingly. When we find a tool has a dependency, the tools labs sysadmins will add it to puppet so that every host part of the grid (current and future) will have that configured accordingly without manual intervention.(source:  http://lists.wikimedia.org/pipermail/labs-l/2013-March/001047.html)What is the labsconsole? [is it important to keep this concept? I think it's worth the sentence below because in some places the docs still refer to "labs console".] https://wikitech.wikimedia.org/wiki/Help:AccessIt 's the old name of what is now Wikitech (wikitech.wikimedia.org)I'm being prompted for a password when I try to 'become my-tool-account'. What's wrong?If you are seeing  "a password is required" message when you try to become your tool (i.e., sudo yourtoolaccount), it is likely because you were logged in to your Labs account when you created the tool account. Unix group membership is checked on login only, so an existing session will not have access to the new tool group. Log out and then log in to your Labs account again to fix this problem.*******************************************************************************************************Moving from Toolserver to Tool Labs:  https://www.mediawiki.org/wiki/Wikimedia_Labs/Migration_Of_Toolserver_Tools   This is a very helpful doc. ...I suggest we collect more toolserver-transition specific help in this document and link to it from Tool Labs Guide...Modifications:Replace "bot project" with "toolsbeta project" (the current name of the project)(modify)  6.9 What's a "service group"? --> Note that Tool Labs terminology is "Tools account"Perhaps clarify:  "service group" is a general Labs concept; "tool account" is what we use service groups for in the Tool Labs project.Additional help topics: 6.18 What should I do about Toolserver links? >  http://lists.wikimedia.org/pipermail/toolserver-l/2013-May/006143.html  (.htacess redirect info)6.19 Anything I should know about Ubuntu Linux?Not familiar with Ubuntu Linux? Below, you can find some good sources of information.You have to do with a Debian-based Linux distribution. Basically if there are different documents you'll probably want to look for the server (not the desktop) edition. The current stable release that has long term support is "Ubuntu Precise", a.k.a "Precise Pangolin", a.k.a. "12.04". This is what's running in Tool Labs.Once you know what a software/command line tool is called you can read the manual directly in Tool Labs by entering "man ".Official Ubuntu Documentation:  https://help.ubuntu.com/Find groups or IRC channels in your language:  http://loco.ubuntu.com/teams/Stuff in German:http://wiki.ubuntuusers.de  - very good explanations of software in German, command line as well as graphical tools. (In contrast, the forum.ubuntuusers.de contains relatively much bullshit.)Please add more links to docs in different languages if you can really recommend them! > Context for question:  http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#End-user-support      > Source for answer: You! 6.20 Can I move my SVN repository?>  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Version_control > http://lists.wikimedia.org/pipermail/toolserver-l/2013-June/006175.html > http://lists.wikimedia.org/pipermail/toolserver-l/2013-June/006177.html6.22 Will Tool Labs have  meta database tables like the ones in Toolserver?> See thread:  http://lists.wikimedia.org/pipermail/labs-l/2013-July/001390.html https://bugzilla.wikimedia.org/show_bug.cgi?id=48626https://bugzilla.wikimedia.org/show_bug.cgi?id=486256.23 Can I join user databases with wiki ones? https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_featuresThat 's actually working today; databases created by users can be joined with the replica databases so long as they are created on the same DB (i.e.: connect to the same place)6.24 Will the commons database be replicated to all clusters, like it is on the Toolserver? https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_featuresIt is not. However, there is a federated database link to commons and wikidata on every shard. There are some caveats in using them for good performance, but they are available. (Needs documenting)6.25 Can I import my JIRA bug tracker information to Bugzilla? If so, how?No, sorry. At the moment, we don't have a solution for you to keep the bugreport from jira. Some people have done such a migration but this would have to be thoroughly tested and would probably need a lot of manual intervention.The good news is: If you wish you can request to have your own "component" in bugzilla for supporting your tool.>  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bug_tracker > any other suggestions?6.26 Does the Tool Labs grid support qcronsub?> https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#qsub_et_alNot directly. Similar functionality is available with the -once option to jsub but the common use case of "continually try to restart my tool if it failed" is better served with the support for continuous applications (-continuous, jstart)Can I move a multimaintainer project to Tool Labs?> https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuffThe short answer; all Tool Labs tools are "multimaintainer" as meant on toolserver (i.e.: shared project space, group membership)My bot has many maintainers. How should we handle credentials?>  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuffCredentials the bot itself uses to edit/perform actions is subject to the target wiki's policies on sharing credentials and may need to be cleared with them.Does Tool Labs support all the packages Toolserver does? > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bots_and_webservices_project  (Is this yes? or were requested packages dif. from total ones?)      >  https://wiki.toolserver.org/view/User:Dab/Debian-Packages      > https://jira.toolserver.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+TS+AND+%28summary+~+install+OR+description+~+install+OR+comment+~+install%29+AND+issuetype+%3D+Task+AND+resolution+%3D+FixedIt 's a "yes" insofar as "If you need it and it is open source, we can get it for you." We have not, on the other hand, preemptively every version of every dependency that has been enumerated; but do so on demand (partly to avoid installing obsolete packages that are no longer relevant).*******************************************************************************************************INVENTORY OF CURRENT TOOL-LAB RELATED DOCUMENTATION, AND PROPOSED UPDATE/REDIRECT PLAN: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help  [UPDATE/REPLACE WITH NEW, EXPANDED GUIDE OUTLINED ABOVE] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools  [UPDATE WITH HIGH-LEVEL OVERVIEW/INFO ABOUT CREATING ACCOUNT] Sources for expansion:  :http://www.mediawiki.org/wiki/File:Tool_Labs_presentation_(Hackaton_2013).pdfhttps://wikitech.wikimedia.org/wiki/Main_Page  [UPDATE:  Under Labs/add note about Tool Labs and add link to Tools project and Tool Labs guide]  https://wikitech.wikimedia.org/wiki/Help:Terminology  [UPDATE with additional terms--Tool Labs, Toolsbeta project, Tools project, grid, cluster... other terms?] http://tools.wmflabs.org/  [UPDATE to clarify what these links are (e.g. "Request access to Tool Labs"; add additional context at top of page to explain what is on page] https://www.mediawiki.org/wiki/Wikimedia_Labs  [Appears current/recently updated] https://www.mediawiki.org/wiki/Developer_access  [UPDATE with context-- what does 'developer access' mean exactly? ] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features [update]https://wikitech.wikimedia.org/wiki/Help:Getting_Started  [Appears current] https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta  [UPDATE with more info about Toolsbeta and how to get access] https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Project_storage  [UPDATE: there's a TODO in the page] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools  [Update: replace 'bot' with 'toolsbeta'/also add content generated/organized in sprint] https://www.mediawiki.org/wiki/Wikimedia_Labs/TODO  [Appears current] http://www.mediawiki.org/wiki/Toolserver/List_of_Tools  [Appears current]    http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design  [Appears current] https://wikitech.wikimedia.org/wiki/Help:Shared_storage  [appears current]  https://www.mediawiki.org/wiki/Special:PrefixIndex/Wikimedia_Labs/: https://www.mediawiki.org/wiki/Wikimedia_Labs/Authentication_improvement_project  [Appears current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Instance_creation_improvement_project  [Appears relatively current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ  [recently updated, but would love to redirect this to guide/combine info there] https://www.mediawiki.org/wiki/Wikimedia_Labs/Create_shared_sql_service_for_all_projects  (last updated jan 2012) Redirect to roadmap/ https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Per-project_saltstack_remote_execution  (last update june 2012) Redirect to roadmap/ https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Shared_home_directories_per_project   (last update dec 2011) Redirect to roadmap/ https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/TODO  (Appears current) https://www.mediawiki.org/wiki/Wikimedia_Labs/Things_to_fix_in_beta  (Last updated June 2012) Redirect to roadmap/  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/List_of_Toolserver_Tools/tool  [Appears current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Log_bot_for_wikimedia-labs_channel  (last update Dec 2011) Redirect  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Nagios_management_without_exported_puppet_resources  (last update Dec 2011) Redirect  to roadmap/ https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en  [Update/to replace 'bots' with 'toolsbeta' to avoid confusion] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Rules  [Update with rules] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs  [Update overview at top of page] https://www.mediawiki.org/wiki/Wikimedia_Labs/Reverse_proxy_for_web_services  [last update march 2012/redirect to roadmap] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design  (appears current)  https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/List_of_Toolserver_Tools  [appears current]