Wikimedia Engineering/Report/2011/February

Major accomplishments this month include:
 * the build-out of our new datacenter in Virginia
 * the (painful) deployment of MediaWiki 1.17 to all Wikimedia wikis

Recent events

 * Data Summit (February 4, California) — A lot of fruitful discussions happened during this working session. Notes are available from the working groups on parsers, structured data and analytics.
 * FOSDEM 2011 (February 5-6, Brussels, Belgium) —
 * GNUnify 2011 (February 11-12, Pune, India) —

Upcoming events

 * Berlin Hackathon 2011 (May, Berlin) —
 * Wikimania (August 2-6, Haifa, Israel) — This year's Wikimania will be preceded by two days of hacking. Mark your calendar for August 2-3! You can also submit a talk or workshop for the Technology tracks of the actual conference (August 4-7).

Hiring
Are you looking to work for Wikimedia? We have a lot of hiring coming up this year, and we really love talking to active community members about these roles. The following positions are currently open:
 * Volunteer Development Coordinator
 * Performance Engineer
 * Software Developer (Features)
 * Software Developer (Mobile)
 * Data Analytics Engineer
 * Operations Engineer
 * Senior QA Engineer

In addition, we hope to post the following positions over the next few months:
 * Release Engineer
 * Technical Writer
 * Network Engineer (contractor)

Operations
Virginia Data Center — Installation of a world-class primary data center for Wikimedia Foundation websites.
 * Status: All the hardware has been delivered, and our team (Rob Halsell, Ryan Lane, Priyanka Dhanda and Ariel Glenn) has been building it out.
 * Program manager: Mark Bergsma

Media Storage — Improvement of our media storage architecture to accommodate expected increase in media uploads.
 * Status:


 * Program manager: Mark Bergsma

Monitoring — Operations and public monitoring system to improve overall uptime, prevent outages, increase transparency and support progress tracking.
 * Status:


 * Program manager: Mark Bergsma

Virtualization cluster — Environment to deploy temporary machines for testing and experimentation, for use by WMF staff and volunteers working on important projects (as capacity allows).
 * Status:


 * Program manager: Mark Bergsma

Backups — Improvement of backup coverage of Wikimedia-hosted data.
 * Status:


 * Program manager: Mark Bergsma

Data Dumps — Improvement of processes to create and provide public copies of public Wikimedia data.
 * Status: Data dumps have been put on hold until the new data center is up and running.
 * Program manager: Mark Bergsma

Content Quality and Editorial Tools
Article Feedback — A feature to collaboratively assess article quality and incorporate reader ratings on Wikipedia.
 * Status: The deployment to our prototype has surfaced bugs that we've fixed. Now that MediaWiki 1.17 has been successfully deployed, we can release the latest version of the Article feedback tool on the English Wikipedia, as part of our pilot experiment . Requirements for the next version (3.0) are being drafted.
 * Program manager: Alolita Sharma

Pending Changes — A feature to allow changes made by logged-out and new users to be reviewed before they appear as the primary version of an article.
 * Status: Developer Aaron Schulz has focused on bug fixes. Further development is waiting for the English Wikipedia community to come to a consensus regarding what the future of the trial should be. A new Request for Comment was started for this purpose.
 * Program manager: Alolita Sharma

Controversial content management system — A feature to handle controversial content on a wiki.
 * Status: Following the 2010 Wikimedia Study of Controversial Content, Brandon Harris has created mockups of the feature, including initial UI design recommendations, in collaboration with the Community department and Board member Phoebe Ayers, who also sent an update. They will be presented to the Board of Trustees by the Strategic product team
 * Program manager: Alolita Sharma

External review system — An interface for external reviews of Wikipedia content.
 * Status: At the request of the Strategic Product Department, Guillaume Paumier has researched and compared previous and current initiatives of quality review of Wikipedia content. He has also analyzed the goals and needs of both Wikipedians and "experts", in order to publish a set of requirements for an extensible and flexible external review system.
 * Commissioned by: Erik Möller

Discussions and Interactions
Liquid Threads — A feature that brings threaded discussions capabilities to Wikimedia projects and MediaWiki.
 * Status: Andrew Garrett has published documentation on upcoming back-end and architecture changes. New design specifications have been published by Brandon Harris as well. A discussion was started on the "gender gap" mailing list about how this new discussion system could improve interactions between participants.
 * Program manager: Alolita Sharma

SimpleSurvey/2.0 — A feature to create and run surveys in MediaWiki.
 * Status: Trevor Parscal has been evaluating the existing codebase, refactoring the SimpleSurvey extension, and consolidating code from other survey extensions.
 * Program manager: Alolita Sharma

Multimedia Tools
Upload wizard — A feature that provides an easier way of uploading files to Wikimedia Commons, the media library associated with Wikipedia.
 * Status: Ryan Kaldari has joined Neil Kandalgaonkar to fix bugs and prioritize the work to be done for an UploadWizard 1.0 release.
 * Program manager: Alolita Sharma

JavaScript parsing library — A JavaScript parsing library for wikitext.
 * Status: Neil Kandalgaonkar implemented a JavaScript parser for wikitext using Parsing expression grammar. It will allow JavaScript tools to support internationalization, templating and other features; it will especially benefit multimedia and Media labs tools. Integration with ResourceLoader is underway.
 * Program manager: Alolita Sharma

MediaWiki infrastructure
Resource loader — A feature to improve the load times for JavaScript and CSS in MediaWiki, enabling faster loading of the Vector skin, media extensions, and anything else that makes extensive use of Javascript and CSS.
 * Status: The deployment of MediaWiki 1.17 to Wikimedia sites has surfaced many bugs. Roan Kattouw and Trevor Parscal have worked on fixing them, and were also available for an IRC office hour to help JavaScript maintainers fix compatibility issues.
 * Program manager: Alolita Sharma

Wikimedia Labs
HTML5 media projects — A set of features to improve media handling and key infrastructure support tools, many developed with Kaltura, such as Metavid, MwEmbed, and the Video Editor.
 * Status: Michael Dale has been working on the integration of TimedMediaHandler and the Add MediaWizard with the Resource loader.
 * Program manager: Alolita Sharma

MediaWiki development
MediaWiki 1.17 deployment — Deployment of the latest MediaWiki version (1.17) to Wikimedia sites.
 * Status: In preparation for the planned deployment of MediaWiki 1.17 on February 8, all outstanding revisions were reviewed. The deployment was attempted twice that day, and eventually postponed because of major performance issues that caused an outage. The problems were investigated, and another plan was published, based on heterogeneous deployment (meaning not all wikis would run the same version of the software). Tim Starling and Roan Kattouw developed wmerrors, a PHP extension to display fatal error pages for PHP. On February 11, a first wave of small wikis were switched to MediaWiki 1.17. On February 16, other small and medium-sized wikis were switched. An attempt to deploy to our biggest wiki (en.wikipedia.org) resulted in a short outage. The English Wikipedia and all remaining wikis were successfully upgraded to MediaWiki 1.17 later that day. Many issues encountered this month were due to the large amount of code changes involved. In the future, software deployments should be smaller and happen more regularly, hence reducing the risk of repeated outages.
 * Program manager: Rob Lanphier

MediaWiki 1.17 release — The upcoming MediaWiki release.
 * Status: Now that MediaWiki 1.17 has been deployed to all Wikimedia wikis, remaining bugs are expected to surface and be fixed. MediaWiki 1.17 should officially be released in March for third-party users (see draft release notes). Its main feature will be the Resource loader. It will also include category collation improvements. Developers are already discussing MediaWiki 1.18.
 * Program manager: Rob Lanphier

Test framework deployment — Creation of an automated test environment for MediaWiki using CruiseControl, Selenium, and PHPUnit.
 * Status:


 * Program manager: Rob Lanphier

Technical Documentation – Improvement of our technical documentation by making small, incremental improvements to the docs and docs process.
 * Status:


 * Program manager: Rob Lanphier

Wikimedia analytics
udp2log — A custom data analytics logging system.
 * Status:


 * Program manager: Rob Lanphier

OWA — Installation and customization of an Open Web Analytics (OWA) platform to process data to support decision making
 * Status:


 * Program managers: Rob Lanphier & Tomasz Finc

Fundraising
2010 Fundraiser — Engineering support for the yearly fundraiser (includes fraud prevention, CentralNotice, and the analytics upgrade).
 * Status:


 * Program manager: Tomasz Finc

Mobile
Mobile site rewrite — Port of our existing gateway to another framework for easier support & collaborative development.
 * Status:


 * Program manager: Tomasz Finc

Offline
Wikipedia version tools — Support and development of a series of tools to select Wikipedia content for offline use.
 * Status:
 * Program manager: Tomasz Finc

OpenZim integration — Support and development of a standard file format for offline Wikimedia content.
 * Status:
 * Program manager: Tomasz Finc

Kiwix UX study — Evaluation of the user experience of the Kiwix mobile app to access offline Wikimedia content.
 * Status:
 * Program manager: Tomasz Finc