Wikimedia Labs/Tool Labs/Migration of Toolserver tools

= Information about design, functionality and structure of Tool Labs for existing and new devs =


 * Labs: a cloud-based infrastructure for developing and running tools run by Wikimedia Foundation
 * Labs vs. Tool Labs explained

= Contact persons at WMF and WMDE =

Wikimedia Foundation

 * Ryan Lane - Labs Operations Engineer, Timezone UTC-7
 * Marc A. Pelletier - Tool Labs Operations Engineer, UTC-4
 * Sumana Harihareswara - community liaison/documentation/workshops, Timezone UTC-4

Wikimedia Germany

 * Silke Meyer - project management @WMDE, coordination of the transition process, Timezone UTC+1)
 * Johannes Kroll - software engineer, technical support for migrating projects, Timezone UTC+1
 * Lydia Pintscher - community liaison/communication, Timezone UTC+1 (from April 2013 on!)

= Mailing list and IRC =

The central places to get help and to meet the crowd are
 * 1) labs-l, the mailing list. This is an excellent place for your questions. And subscribe to it too!
 * 2) #wikimedia-labs connect, the IRC channel on the freenode network (more information).

= Table of features and tools' dependencies needed for current tools =

At the end of 2012, a list of features has been started. Feel free to add things! In the first part, we are collecting indispensable technical features Labs needs to offer what the toolserver has to offer now. Below, there is a list of features that are nice to have.

Wikimedia Labs/Toolserver features wanted

'''In addition, we are collecting technical details (like software and data dependencies) about tools on the toolserver to make Tool Labs fit for the existing tools. We need all tool developers' help to complete that list.''' Please edit it here:

Toolserver/List_of_Tools

List of available features
Ryan's list (end of October 2012)


 * 1) A compute service where users can create virtual machines (instances) in a self-service way inside of projects they manage
 * 2) A shared storage service that works in a per-project or global (read only) manner
 * 3) A networking service which enables users to manage floating IP addresses, DNS entries, and firewall rules
 * 4) A puppet service which enables users to manage puppet configuration in a per-project manner
 * 5) (planned - high priority) a saltstack service which enables users to manage various services within their projects; for instance, a way to manage bots via an API, or a Database as a service
 * 6) (planned - high priority) a database service, to act as the equivalent of Toolserver's user databases. Requires saltstack service.
 * 7) (planned - high priority) Replicated databases.
 * 8) (not currently planned - low priority) Queue as a service. This could be used as a replacement for sun grid engine, but this would take some engineering work.
 * 9) (not currently planned - low priority) Block storage as a service. This could be used to mount extra space dedicated to a single instance, when that instance need storage that looks like disks.
 * 10) A monitoring service which acts in a per project way. We have both nagios and ganglia that monitor all instances. Projects are broken into clusters.
 * 11) (planned - low priority) A logging service that acts in a per-project way and enables users to search their instance and application logs.
 * 12) A way to handle sudo across instances in a per-project manner.
 * 13) Centralized authentication and authorization across all instances and projects (and most services)
 * 14) Centralized and automated documentation for project resources (wikitech.wikimedia.org)

= Schedule for the transition =

Road map for upcoming features
The road map for the availability of features in Labs is due in May to be presented to WMDE's members' assembly. Details are provided by WMF staff with the help of WMDE. This road map will clarify when the functionalities needed for migrating tools from the toolserver to Labs/Tool Labs.

WMF's milestones for the months to come can be found here: Wikimedia Engineering/2012-13 Goals

Steps to prepare and complete the transition
tbd

= List of important questions/FAQ =

Bots

 * wikitech:Nova Resource:Bots
 * wikitech:Help:Move your bot to Labs

Webtools

 * wikitech:Nova Resource:Webtools

More to be added here...

When can I migrate my software to Labs?
It depends on what you are running on the toolserver. If your tools depend on database replication you cannot migrate yet. Database replication will be made available at some point in the 2nd quarter of 2013. User databases are available now. If you are running a bot or a webservice that does not depend on db replicas you can try Labs. Please note that currently Tool Labs is an environment that is partly experimental.

What do I have to think of when finishing migration?

 * Please clean up on the toolserver: Once your tool is running inside Labs/Tool Labs, please delete it from the toolserver and announce to the toolserver admins that your account can be deleted.

Where's the bug tracker?

 * WMF uses using Bugzilla. Please use it for your issues!
 * There are the following categories in Bugzilla:
 * product: "Wikimedia Labs"
 * component "bots" - for bot-related issues
 * component "General" - Issues surrounding set up of labs projects.
 * component "deployment-prep" - Bugs relating to the configuration/management of the WMF deployment-prep project at http://beta.wmflabs.org. More info: wikitech:Nova Resource:Deployment-prep.
 * component "Infrastructure" - Bugs related to Wikimedia Labs infrastructure and not a specific Labs project.
 * component "webtools" - issues related to webtools
 * component "wikistats" - http://wikistats.wmflabs.org/
 * If former Jira users need help / introductions, please don't hesitate to ask your questions on labs-l or on IRC (links above). Inform us about wishes for workshops! We are considering to offer workshops about software you are/will be working with in Labs that you didn't have on the toolserver. Tell us what you need!

Where is documentation?

 * wikitech:Help:Getting Started
 * Labs Glossary

What about file permissions? Who can see my code?
There are projects where users have root, so that all users in the project have full access to the whole project. This setup is not mandatory though: Tools can also use tool user IDs to control file permissions. On the tools project (which will be where toolserver tools migrate), you have full control over access permissions of your code and data. By default, only the tool maintainers have access (all the maintainers of a tool are in the tool's group).

Do stewards have a specific project on WMF Labs?
There is no plan to have distinct projects for different tool makers; but the tools are separated from each other. There is nothing that prevents you from sharing the maintenance of some tools between different stewards (in fact, it is recommended that you do so to ensure that there is always someone able to keep them up at need).

= Future of WP-OSM =

= How can chapters, communities and developers support the process =

Now / Phase of preparations

 * You are running software on the toolserver? Please document what you need for it to work (link above)! Focus on functionalities rather than exact details, e.g. software versions.
 * Please talk to us if you are ready to become an early adopter to try Labs, e.g. once the database replication is up and running.

Know how and documentation

 * You are planning migration or did already accomplish it? Please help us to improve and add documentation. If you miss documentation, poke us. If you found out undocumented processes on your own tell us about them or write them down. If you don't know where to put documentation, don't hesitate to ask!
 * Would you like to give a talk or a workshop on a topic related to the Toolserver -> Labs transition at a Hackathon or online? Don't hesitate to propose it to us!

Hands-on support for others

 * You have migrated your own project successfully and feel like supporting somebody else? You are able to adapt software that needs modifications? Your help would be greatly appreciated! Please talk to us!

Financial support
Are you a chapter or some other project with funding, using the toolserver?
 * Please consider donating for the toolserver so it can run until we have got an alternative ready.
 * If you have paid developers in your projects, please consider paying them to migrate your project from the toolserver to Labs. You could even have them offer support to another / a related volunteer project if this is fine with the volunteers.

= Links for further reading =

Labs

 * 

Technical discussion on MediaWiki.org

 * Labs