Wikimedia Engineering/Report/2013/April

Engineering metrics in April:
 * 116 unique committers contributed patchsets of code to MediaWiki.
 * The total number of unresolved commits remained stable around 815.
 * About 49 shell requests were processed.
 * Wikimedia Labs now hosts 163 projects and 1224 users; to date 1782 instances have been created.

Major news in April include:
 * the start of recruitment for a multimedia engineering team;
 * the deployment of a better translation interface;
 * the release of Kiwix of Android, an app to download and view Wikimedia content offline;
 * the milestone of 500 million monthly unique visitors reached in March;
 * improvements to the main page of translatewiki.net;
 * the migration of Wikidata, and the English and German Wikipedias, to MariaDB 5.5;
 * the second phase of Wikidata, whose structured data can now be displayed in Wikipedia articles;
 * the deployment of VisualEditor's alpha version to 14 more language versions of Wikipedia;
 * a proposed replacement for the login and account creation interface on Wikimedia projects;
 * the launch of the Language Mavens program, an advocacy and advisory body in the domain of language engineering;
 * the ramp-up of technical mentorship programs;
 * the launch of an official Wikimedia Commons app for iOS and Android.

''Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.

Upcoming events
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.

For a more complete and up-to-date list, check out the Project:Calendar.

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

Announcements

 * Monte Hurd joined the Mobile engineering group as Software Engineer in the Apps team (announcement).
 * Brandon Black joined the Operations team as Dev/Ops Engineer (announcement).
 * Erik Bernhardson joined the Features team as Features Engineer (announcement).
 * Nischay Nahata joined the Features team as Features Contractor (announcement).

Technical Operations
Site infrastructure
 * Several large projects were migrated to MariaDB, with positive results. A new class of redis servers were deployed in support of the migration of our asynchronous job queuing infrastructure from MySQL to redis, enabling us to better meet the demands of Wikidata and Echo.  New file uploads are now being written to Ceph in eqiad in addition to Swift in pmtpa, in support of a potential migration. The current plan is to open up the eqiad Ceph cluster for 'reads' the coming week. Currently 'reads' are served by the Tampa Swift cluster.
 * With the core cluster migrated to Eqiad now, we are now working on the miscellaneous server cluster. For status, please refer to https://wikitech.wikimedia.org/wiki/Tampa_cluster. As part of the cleanup, we retired a bunch of the servers as well.

Fundraising
 * Deployed public reporting host. Upgraded payments-wiki and fundraising-wiki from 1.20 to 1.22. Puppetized and migrated banner log collection pipeline from pmtpa to eqiad. Upgraded Fundraising CiviCRM/Drupal instance, and the instance used by Global Education as well.

Data Dumps
 * New tools for import of partial or full content into a new wiki have been released. A step-by-step walkthrough of their use has been added to the documentation for users of the dumps on Meta.
 * True incremental dumps are now a GSOC proposal and several students have applied for this project.
 * The logging table XML dump on Wikidata was taking days to run, due in part to the high volume of edits there, much more than even the English Wikipedia. Most of those edits wind up being recorded as autopatrol in the log, making it already about half the size of the logging table for the English Wikipedia. Breaking up the database query into smaller batches works around the issue.

Wikimedia Labs
 * Work on tool labs is progressing nicely. 32 bots/tools have been added to the tools project. Most of the functionality of Toolserver should now be available in tool labs. Database replication is still being worked on, but is progressing well. The pre-labs replication databases are being replicated to and the redactatron application has been finished, allowing us to mark tables as ok to replicate. Our current roadmap is for db replication to be accessible by the Amsterdam Hackathon. Instance creation performance greatly improved this month by replacing the generic ubuntu cloud images with our own custom images that pre-installs and pre-configures most of what an initial puppet run would handle. Work on single-instance MediaWiki continued this month, making the initial MediaWiki installation more robust and handling a number of legal issues (such as showing the terms of use, using proper logos, linking to a proper privacy policy, etc.). Work began on adding ajax interactivity to the OpenStackManager interface. Currently changes are in for reboot and get console output actions for managing instances. A more reasonable project filter change using jquery chosen has been added as well. Work on replacing glusterfs is mostly done. Two projects have been switched to use the new NFS server and the rest will be switched next month. Work has begun on upgrading OpenStack from the essex release to the folsom release. Our testing environment has been upgraded and production tests are currently ongoing. During the OpenStack summit, work was done to push the Moniker DNS application into OpenStack incubation to be added as a supported OpenStack project. Ryan Lane gave a talk during the OpenStack summit about the state of OpenStack's user committee along with Tim Bell of CERN and JC Martin of eBay. Work on the user committee is in hopes of making OpenStack easier to use an upgrade, which should increase the frequency of updates in Labs.

Kiwix
The Kiwix project is funded and executed by Wikimedia CH.


 * In April we have released for the first time Kiwix for Android, see our blog post for more details. This port doesn't provide as many features like the desktop one, but it works well with all ZIM files. Two developers of Kiwix will also be at Wikimania and we will do a small hackathon, two presentations and try to have a booth; Organisation of these activities has started.

Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.


 * The team hit a big milestone with the deployment of the first iteration of phase 2 of Wikidata on all remaining Wikipedias (it had been enabled on 11 Wikipedias previously). Qualifiers were also enabled on Wikidata, making it possible to add additional information to certain data. Wikipedians are now able to make use of the data available on Wikidata in articles, allowing the data to be collaboratively collected, curated and used by all Wikipedias.
 * The team also fixed a few issues to make it possible to use Wikidata with Internet Explorer 8, and worked on the time datatype. Together with bot owners, they massively improved the time it takes for Wikidata changes to show up in the recent changes and watchlists on Wikipedia sites. The code and architecture got an external professional review; the reviewers were quite happy with the quality of the code base and gave useful tips for improvements.

Future

 * The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.