Wikimedia Labs/Tool Labs/TODO

Here is the current list of things which are known to be needed for the Tools project, with notes and status. In no particular order, and probably incomplete at this early stage.

To clarify, a tool here means a single, self-contained bot, webtool, or other system and all its dependencies. They are the unit of management, each having its own (set of) maintainer. The project is the Labs project where all the tools are (there are two, at the moment, "tools" and "bots").

Feature / Requirement	Implementation	Dependencies	Status / Comments
Scheduling and job management	Open Grid Scheduler[1] is a current candidate		Done The grid is working and fully functional, with easier-to-use tools for the typical cases as well.
Unification of labs project for tools	The final implementation is to deprecate 'webtools' and use 'bots' as the "experimental/flexible" development project, and "tools" as the official Toolserver-replacement home for stable tools.		Done [2]
Tool management	Unix groups, shared storage		Done Tool management is accomplished through the new service group interface of Wikitech. Tool Labs user can create and manage tool accounts without sysadmin intervention (including managing the list of maintainers).
Database replication		General Labs replication	Done, and there was much rejoicing. Yeay.
Tool/User databases			Done They currently live on a project-local instance until database replication is available, however. They will be transparently moved to the central database once that becomes available.
Tool status / monitoring	(Probably via the general management web interface)		Done The Tool Labs landing site links to tools' web interfaces, where they can post status information.
Software dependencies	Distributed on all execution nodes via puppet		Done There is now a well-defined procedure to add software packages to the execution and/or dev environment, including packaging from non-Ubuntu sources.
Bug/Issue tracker	Bugzilla.		Done There is now a Tool Labs tools bugzilla product where maintainers can request a component for any of their tools they want to support that way.
Anonymized weblogs			Done The current webproxy setup anonymises by default. Access and php error logs are split per-tool; there is a limitation in the current Apache which prevents proper splitting of error logs, however. Some mechanism to inspect them by tool maintainers should be looked into.
Local email	Forwarded email addresses What fqdn?	Check on legal / policy / requirements	In progress Per-user and per-tool email addresses (the latter exploding to the tool maintainer or to a mailing list); Legal prefers no local mail storage and is looking into the appropriateness of giving out @tools.wmflabs.org addresses. (E.T.A. October 31st, 2013)
Render		Requires more detail on specific requirements	Done May need to have its own project; discussion with the Labs infrastructure team will be needed
Access to revision text		Look into feasability	N Not done The resources to replicate the revision text are prohibitive. That said, using the API to get revision text from the labs is likely to be faster than direct access to the database given caching.
Shared storage	Now uses NFS	Provided by the infrastructure	Done, gluster has been replaced with a NFS server with redundancy at the hardware level.
Cross-tool authentication	OAuth most likely.		E.T.A. late 2013 (improvement/new feature) Best case: SUL as OAuth server Actually plausible: an OAuth server with 'SUL account name' and 'email' (for instance) as protected resources that tools can request access to. OpenID is soon to be deployed to allow for identification of end-users to the tools themselves.
Database creation from tools			Done Service groups have the privilege to create databases.
Convert PIP packages to debian using some automatic tool	Probably create a tool (c++ / shell?) which automaticaly convert the pip package to debian		Done ssh tools-dev pypi-install <name> --keep check the /tmp for output