Wikimedia Labs/Tool Labs/Needed Toolserver features

See also ../Toolserver features wanted in Tool Labs.

Database

 * MySQL-access to the WMF-project-databases (minimum same table/row/field-scope like on the toolserver).
 * Separate MySQL-server(-cluster) for user- and project-databases.
 * 1 MySQL-Server per cluster (s1-s7) where user- and project-databases can be created.
 * Minimum 1 MySQL-Server per cluster (s1-s7) where a live-copy of commons is running in parallel.
 * Minimum 1 MySQL-Server per cluster (s1-s7) where a live-copy of wikidata is running in parallel.
 * Support for short- and long-running queries (maximal run-time depending on the load up to few days).
 * Mail-message-system about killed queries.
 * If the labs system is not as underpowered as toolserver, in terms of the database servers, a query killer may not be necessary. CBM (talk) 21:54, 20 December 2012 (UTC)
 * DNS-aliases for clusters (sql-s1.toolserver.org) and wikis (enwiki-p.rrdb.toolserver.org).
 * replication-replag-reporting-system.

Filesystem

 * home-directories. ✅
 * project-directories for working together. ✅
 * common-directory for wikimedia-dumps with an automatic update of them. ✅
 * common-directory for page-stats with an automatic update of them.
 * Quota limits on filesystem usage / method for users to check their quota usage

Languages

 * i18n for all (available) languages of this world.
 * Can you clarify this? i18n for what specifically?--Ryan lane (talk) 21:06, 20 December 2012 (UTC)
 * Maybe this is a reference to the [//wiki.toolserver.org/view/Toolserver_Intuition Toolserver Intuition] framework? – Minh Nguyễn (talk, contribs) 06:38, 21 December 2012 (UTC)
 * Support for perl, c, python, c++, php, mono, java, tcl, bash-, ksh- and zsh-script (for (fast-)cgi and cli). ✅
 * Anything that comes with Ubuntu, or someone is willing to package (that is open source) is available.--Ryan lane (talk) 21:06, 20 December 2012 (UTC)
 * Tons of libraries for each programming-language.
 * Same as previous.--Ryan lane (talk) 21:06, 20 December 2012 (UTC)

Web

 * Bugreporting-site (issue and bug tracker like JIRA or bugzilla) with the possibility to create (sub-)projects that can administrate by an user/project.
 * preferably with conversion/migration from TS JIRA to the new one on labs
 * SVN Git repositories ✅ (people can already request Git repositories in Gerrit for labs stuff)
 * Why SVN? -- Krenair (talk &bull; contribs) 21:05, 20 December 2012 (UTC)
 * Agreed. Let's switch everything to git and be done with svn once and for all.--Ryan lane (talk) 21:07, 20 December 2012 (UTC)
 * To clarify, I'm fine with people hosting their code in SVN. What I'm saying is that WMF won't provide a SVN hosting solution. If volunteers want to create an svn project in Labs, and run svn.wmflabs.org, that's fine.--Ryan lane (talk) 21:52, 20 December 2012 (UTC)
 * Access to (anonymized) web- and web-error-logs.
 * Server statistics, workload, status and else
 * E.g. Munin or something similar, please confer TS Munin
 * General server (service) status-page on a external and independent system with own password-system (human and machine-readable e.g. XML, JSON, csv), please confer TS Status
 * May be webserver page view/visit statistics too
 * https://graphite.wikimedia.org/ (requires labs account)
 * https://gdash.wikimedia.org/ (open to the world)
 * (more at http://old.nabble.com/What-is-slow-td34077918.html)
 * Server/project for web tools and pages. ✅

Job-system

 * Job-system for executing of non-web-tasks, e.g. stable cron.
 * crontab or else without "(CRON) CAN'T FORK (child_process): Not enough space." issue, please confer Cron#Timing and work load
 * may be add a queue system like SGE
 * Various resources like run-time, needed databases, free memory and user-slots.
 * Automatic switch-over and moving in case of a server-failure.
 * Possibility to let a non-root maintain the job-system (adding resources, killing jobs, etc. pp.).

OSM

 * A Postgresql-server for OSM.
 * Render-server with enough space to host several layouts.


 * OSM is being added to WMF production. We'll have a test/dev version of OSM in Labs, but we have no plans on having a quasi-production like version of OSM in Labs.--Ryan lane (talk) 21:08, 20 December 2012 (UTC)
 * Without a live OSM-planet-db the labs fails the criteria to replace toolserver. --Kolossos 15:20, 23 December 2012 (UTC)


 * We have special projects on toolserver like WIWOSM that combine Wikipedia- and OSM-data in one script. This is only possibble if a script can connect to both databases. Without such an feature we will lose this service in Wikipedia. --Kolossos 18:02, 1 February 2013 (UTC)

Backup

 * Daily backups for user- and project-directories for minimum 1 week.
 * including crontab and all other necessary configuration files
 * Daily backups for user- and project-databases for minimum 1 week.

Various

 * Mail-address for users.
 * Mail-address for projects.
 * Configurable Mail-forwarding for both.
 * Nagios-system. ✅ (see: http://nagios.wmflabs.org/nagios3/)
 * We have a ganglia system too; see http://ganglia.wikimedia.org/latest/ --Ryan lane (talk) 21:15, 20 December 2012 (UTC)
 * Nothing to see, password protected. – Giftpflanze (talk) 15:50, 23 December 2012 (UTC)
 * It's temporarily private - see wikitech-l/2012-December/065184.html. -- Krenair (talk &bull; contribs) 16:11, 23 December 2012 (UTC)
 * Free license-choice of the user for his/her tools.
 * This is of course the case, as long as the license is an OSI approved license.--Ryan lane (talk) 21:09, 20 December 2012 (UTC)

End-user-support

 * 1 or more root with an understanding for user-problems.
 * We have 3 on-staff and many volunteers.--Ryan lane (talk) 21:10, 20 December 2012 (UTC)
 * Having a root who is *understanding* is particularly important. Many users are not experts in what they are doing, and need friendly help in solving problems or cleaning up messes they have created
 * Documentation for beginners and non-technicals.