Wikimedia Labs/Toolserver features wanted in Tool Labs

Below is a list of features of toolserver which would be cool to have available on the projects that are supposed to replace toolserver:

Labs wide (not only bots / tools), but available for all projects
seem to include database database replication in a form that makes it a useful, direct replacement for toolserver]]».
 * Access to production db (read-only / replicated) (high priority)
 * joining of user databases with wiki databases
 * will the commons database be replicated to all clusters, like the toolserver?
 * allow queries like the one this researcher was desperate for
 * Unlikely, that logic should be handled within the application. It's impossible to shared data otherwise, as well as the extra overhead on the database servers which are effectively a shared component. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * Explanation by Carl (CBM): «[[mailarchive:toolserver-l/2012-September/005382.html|plans for WMF Labs don't
 * Home directories (done?)
 * What is meant by this? Labs is meant to be a collaborative, community maintained environment. I specifically want to avoid the Toolserver way of individualizing everything. If a user leaves, their bot, or tool, should very easily be transferable to another user. We tend to opt for using project storage (/data/project) rather than home directories. It's also preferable to run things as service users rather than individualized users, though that isn't a requirement.--Ryan lane (talk) 22:17, 25 September 2012 (UTC)
 * I should also mention that we already have per-project home directories, where the directories are accessible to every instance in a project. It's possible to do things the toolserver way, but it's a very limiting way of doing things.--Ryan lane (talk) 22:17, 25 September 2012 (UTC)
 * Lets cross this, it isn't really relevant in labs. Sure every ssh user has a home directory, that's standard Linux. But for storing applications and databases, this must be stored elsewhere (on the mounted project storage, not in a home directory, not on the instance itself, he instance itself needs to be recyclable from puppet). Krinkle (talk) 02:42, 26 September 2012 (UTC)
 * Looks like a bad argument. You wrote in mailing list that things can be abstracted etc., so let's not strike things before the users actually get what they expect and need. I guess this point means "provide a way to make stuff available which is as easy as placing a file (executable or not) in public_html"? --Nemo 05:15, 26 September 2012 (UTC)
 * Right, I meant to mark done, not strike. This is done. Krinkle (talk) 06:46, 26 September 2012 (UTC)
 * Technically home directories are done. They've been available since Labs launched. We discourage their use fairly heavily, though. Ideally, the only thing that should go into a user's home directory is their environment settings. Home directories are personal, and therefore they explicitly stop collaboration; people are generally unwilling to go into another user's home directory, even if the user retires or disappears. Rather than using home directories, users should use project storage, which is shared to all instances in a project, just like home directories. We encourage project storage to be used in a fairly open way (not per-user, but per-bot or per-tool, or per-subproject).--Ryan lane (talk) 06:09, 26 September 2012 (UTC)
 * Central per-user directory mounted on all instances within a project (done)
 * Again. Let's try to avoid per-user things. We have shared storage at /data/project that is accessible by everyone in a project. It's possible to lock this down by file permissions, even to per-user, but it's way better to make things owned by a service user (and control access to that user), or to have things owned by the project group.--Ryan lane (talk) 22:18, 25 September 2012 (UTC)
 * Can someone explain the difference between this and home directories?--Ryan lane (talk) 22:18, 25 September 2012 (UTC)
 * If the previous point means what above, this is probably about being able to access user data? Is there a way on Labs to easily share data across all projects? --Nemo 05:15, 26 September 2012 (UTC)
 * As mentioned above, yes, there is per-project storage that is shared to all instances in a project. It's accessible at /data/project.--Ryan lane (talk) 06:09, 26 September 2012 (UTC)
 * Backup of home directories and user databases
 * Backups of databases should likely be handled by users, and saved in project storage--Ryan lane (talk) 06:11, 26 September 2012 (UTC)
 * It's possible that we'll do automatic backups of the user databases as well, assuming they are provided by a database service.--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * Create generic project for web tools (Bug 40510) (done)
 * Create generic project for periodic/long-running bots (done)
 * It is not clear yet whether people should share instances or create their own. As they are unlikely to interfere with each other, the overhead of N linuxes may not be worth it. Instead it may be more useful to have 1 big instance, or a grid of instances but control distribution with SGE instead of manually.
 * 'production' tool labs should be locked down access wise and run with some form of scheduler, development should be shared instances for collaboration. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * Agreed. We could use SGE for this, but it really seems like overkill...--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * Is there a "standard" way of job scheduling in Ubuntu shops? --Tim&#160;Landscheidt 12:27, 8 October 2012 (UTC)
 * Local and auto-updated copies of:
 * Wikimedia XML Dumps (done)
 * This is accessible on every instance at /public/datasets
 * visits per page (pagecounts) and project (projectcounts)
 * same visits stats, in database format to allow direct querying
 * Simple setup to allow HTTP access to projects/instances (reverse proxy, port forwarding, public ip)
 * A http proxy is on the todo list for labs. Some projects have shared web instances currently. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * Misc. Toolserver features:
 * Support SGE to automatically defer starting of expensive processes based on current capacity and usage (qcronsub, qsub) https://wiki.toolserver.org/view/Job_scheduling#arguments_to_qsub/qcronsub
 * If the presented alternative has the same user interface, it shouldn't be a problem. For instance, people don't have an opinion about which of the SGE forks would be preferable.
 * WikiMiniAtlas depends on:
 * the OSM database mirror being available
 * the WIWOSM project (although dschwen could proxy that from the TS)
 * Proxying is not allowed per our terms of use. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * If we're looking to replace TS, then we need to make this available.--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * Dispenser's coordinate extraction database (GHEL)
 * Replicate or transfer MMP (multimaintainer projects) from Toolserver
 * Ryan says: "They can be created in LDAP by making a labsconsole account. Additionally, unless the account needs to directly log in via ssh, there's no request process needed for the user to be used. Alternatively, the user could be created as a system account via puppet."
 * What's that supposed to mean? The migration of (multi-maintainer) projects is the goal of this list, not one of its means.  --Tim&#160;Landscheidt 12:27, 8 October 2012 (UTC)
 * servlets
 * missing support blockers for migration
 * no support for new users not familar with unix based systems
 * Can someone explain what this means, or how TS is currently doing this?--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * no transparent updating of packages with security problems/bug (done)
 * All instances by default are configured for unattended-upgrades. A general overview/management system has been roughly discussed and would be nice to implement though. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * So are you sure that everything requested under this point is done? --Nemo 11:25, 27 September 2012 (UTC)
 * permanent blockers for migration
 * license problems ("i wrote code at work for my company and reuse parts for my bot framework. I have not the right to declare this code as open source which is needed by labs policy") ❌
 * This isn't going to change, we should be open and collaborative. Restricting access to code goes against that. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * +1. We will only ever allow open source and open content. I honestly wonder how you are legally using proprietary software on TS.--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * no DaB.
 * DaB isn't a feature but a community member, he can't be technically implemented (though his help would be appreciated!). DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * I don't see anything about this page being about technical implementable stuff only. --Nemo 11:25, 27 September 2012 (UTC)
 * SFTP access to instances (done)
 * This is currently available if you have SSH access. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)
 * Email addresses/forwarding
 * Localisation for tools on translatewiki.net
 * Per-project optional svn and/or git repo
 * For "Tool Labs" that is, since in "WikiDev Labs" this is mandatory workflow
 * Should versioning really be optional? Even for tools, is it ever a good idea not to have a repo? I'd rather improve Git/Gerrit usability and integration. Eloquence (talk) 21:07, 25 September 2012 (UTC)
 * I think it would be hard to enforce the use of source control, unless someone was policing things, or if we required it to deploy tools (maybe via git-deploy?). Using the deployment system for this actually may be the easiest way to enforce this.--Ryan lane (talk) 22:25, 25 September 2012 (UTC)
 * Do we want to have a a bare git server for wmflabs (like there svn.toolserver.org), or do it in Gerrit? Or maybe allow any git url so that users can store it where they like (be it gerrit.wikimedia.org, github.com etc.). Krinkle (talk) 02:42, 26 September 2012 (UTC)
 * No, not another server that someone has to maintain. Gerrit or GitHub are fine.  --Tim&#160;Landscheidt 12:27, 8 October 2012 (UTC)
 * Mysql query killer (especially for queries to the replicated wmf wiki databases)
 * On the roadmap - relatively easy change.--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * Need a clear story for access to bot credentials - should ownership of bot accounts be shared between all users of the server, or should each bot account be owned by an individual user? If the latter, how will projects get access to the credentials? Note that there are some bad practices going on that might affect this: some tools contain hardcoded bot credentials, and some people share passwords between their user accounts and bot accounts.
 * Issue and bug tracker like JIRA (or bugzilla)
 * preferably with conversion/migration from TS JIRA to the new one on labs
 * Stable cron
 * crontab or else without "(CRON) CAN'T FORK (child_process): Not enough space." issue, please confer Cron#Timing and work load
 * may be add a queue system like SGE
 * Server statistics, workload, status and else
 * e.g. Munin or something similar, please confer TS Munin
 * general server (service) status, please confer TS Status
 * may be webserver page view/visit statistics too

Bots project

 * Public webserver for logs (done)
 * CGI (perl, python, ...) for helper tools (done)
 * Some packages to install
 * Support for mono (c#), python, svn, git (done)
 * PyWikipediaBot requires python v2.7.2 or greater but less than v3 but python is currently v2.6.5 on bots-1
 * Is this resolved by using Ubuntu precise? If so, bots that need the newer version can be on Ubuntu precise instances.--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * resolved by new distro -- use bots-nr1 and later for these bots Petrb (talk) 07:36, 18 October 2012 (UTC)
 * PyWikipediaBot and all prerequisites for Perl MediaWiki::Bot and ::API should be installed system-wide on Bots instances
 * Yes. We should puppetize this, so that we can track dependencies properly.--Ryan lane (talk) 01:29, 2 October 2012 (UTC)
 * Text editors: vim (probably some others like nano etc.) (done)
 * Shells - bash, ksh, csh, tclsh (done)
 * Libraries - libtcl (done, needs puppetize)
 * Php, java (done, needs puppetize)
 * Perl (needs full cpan upgrade and MediaWiki:: packages) (done, needs puppetize)
 * Why does it need cpan at all? I understand that on Toolserver with its limited admin capacity it's hard to wait for proper installation, but I assume that in Labs there will probably always be a root around who can build a proper package on the fly.  --Tim&#160;Landscheidt 12:27, 8 October 2012 (UTC)
 * Flup (Python WSGI)
 * imagemagick
 * All the packages requested by TS users and now taken as a given; see e.g. tswiki:User:Dab/Debian-Packages and install requests on JIRA – someone make a list
 * Per-project optional custom MySQL databases Done (bots-sql1, -sql2 ...) Petrb (talk) 07:36, 18 October 2012 (UTC)
 * This is on the roadmap. Will likely come some time after replicated databases.
 * Basically similar to the mounted project-wide storage. Not on any individual instance, accessible from within each project instance.
 * Toolserver also has the principle of public databases that can be read from other projects. This is probably something we'd want too, so that projects can build on top of each other.
 * The current concept behind this in Labs is that all databases will be accessible from all instances. Creation/modification/grants/etc. will be handled by sysadmins in the project that owns the database.--Ryan lane (talk) 06:09, 26 September 2012 (UTC)