the racking party at our new data center in Virginia
the Data Summit that happened in early February in California
the release of Editor Trends study data and tooling
the painful, but ultimately successful, deployment of MediaWiki 1.17 to all Wikimedia wikis.
Note: In the past, each "monthly engineering update" has reported on what was accomplished the previous month: the previous "February update" hence reported on what we did in January. In order to avoid any ambiguity, and to be more consistent with the other Wikimedia reports, we're now going to explicitly call them reports of the previous month. This means this "February report" is about what we did in February.
FOSDEM 2011 (February 5-6, Brussels, Belgium) — Arthur Richards and Tomasz Finc attended and engaged in discussions about data, CiviCRM and Drupal. They also gave an overview presentation of the current state of Wikimedia analytics.
Are you looking to work for Wikimedia? We have a lot of hiring coming up this year, and we really love talking to active community members about these roles. The following positions are currently open:
We are also looking for a contractor in the Netherlands who will support the Operations team in designing and maintaining the Wikimedia network(s), and perform on-site work in the data center facilities in Haarlem and Amsterdam.
In addition, we hope to post the following positions over the next few months:
Virginia Data Center — Installation of a world-class primary data center for Wikimedia Foundation websites.
Status: Nearly all hardware has been delivered to the data center. More than 50 pallets of equipment have been unboxed, stacked and installed in the 16 racks by a 4-person team. Almost everything has been cabled, and we are working on the finishing touches, as well as the initial setup of all devices to make them available for management on the network. In March, configuration of the first clusters of servers and services will begin, while we wait for network transport and transit services to be installed.
Media Storage — Improvement of our media storage architecture to accommodate expected increase in media uploads.
Status: Contractor Russell Nelson has installed and deployed Swift on a test cluster of three machines. Some code has been written to integrate Swift with MediaWiki's thumbnail generation, as well as Squid, the caching proxy software used on the "upload" media serving cluster. We are still working on fixing some bugs and doing some preliminary testing, before we can deploy this test setup to serve a small portion of our media and replicated traffic (read more about distributed file storage choices).
Virtualization test cluster — Environment to deploy temporary machines for testing and experimentation, for use by WMF staff and volunteers working on important projects (as capacity allows).
Status: A new OpenStack has just been released, which contains the software features we need. This project was however also delayed due to the build out of the new data center. We expect to have the virtualization test cluster production ready in March.
Backups — Improvement of backup coverage of Wikimedia-hosted data.
Status: We have purchased a dedicated storage solution which will arrive in March, and improve the reliability of part of our data. Once servers in the new data center are online, and our private connection between Tampa and Ashburn is up, we will be able to replicate all data between the two data centers as well.
Data Dumps — Improvement of processes to create and provide public copies of public Wikimedia data.
Status: Dumps were suspended for the upgrade to MediaWiki 1.17, and delayed by the difficulties encountered during its deployment. They are now running again, and their performance is being tested. With the new version, they include the byte length of revisions (a popular request, implemented by Rob Lanphier). In order to facilitate production on a regular and faster basis, as well as reuse, Ariel Glenn is now looking into production of dumps in many small pieces.
Article Feedback — A feature to collaboratively assess article quality and incorporate reader ratings on Wikipedia.
Status: The deployment to our prototype has surfaced additional feature requirements that we've now addressed. Now that MediaWiki 1.17 has been successfully deployed, we can release the latest version of the Article feedback tool on the English Wikipedia, as part of our pilot experiment this quarter. Requirements for the next version (3.0) are being drafted.
Pending Changes — A feature to allow changes made by logged-out and new users to be reviewed before they appear as the primary version of an article.
Status: Developer Aaron Schulz has focused on bug fixes. Further development is waiting for the English Wikipedia community to come to a consensus regarding what the future of the trial should be. A new Request for Comment was started for this purpose.
Review system — An interface for external reviews of Wikipedia content.
Status: At the request of the Strategic Product Department, Guillaume Paumier researched and compared previous and current initiatives of quality review of Wikipedia content. The analysis of goals and needs of both Wikipedians and "experts" led to a set of draft requirements for an open review system for Wikipedia, as well as an API and user interface for quality indicators.
SimpleSurvey 2.0 — A MediaWiki extension to create and run surveys in MediaWiki.
Status: In our work on the Article Feedback tool, we used some functionality from the existing SimpleSurvey extension. In order to make it more robust, Trevor Parscal has been evaluating the existing codebase, refactoring the extension, and consolidating code from other survey extensions. SimpleSurvey will also help us conduct small surveys to support strategic research.
Status: Volunteer developer Junaidpv created the Narayam extension for MediaWiki, which adds input methods for some Indic scripts. Roan Kattouw refactored it heavily in order to facilitate its future deployment. We're planning to create a team in India to continue to work on Indic scripts, as a first step in our efforts to support non-Roman alphabet editors.
In February, the Community and Tech departments started a joint experiment in which engineers are working even more closely with Community department staff. Developers are "embedded" in the Community department, to try out a more agile way to prototype features. Trevor Parscal started in this role in February, and will continue in March.
In February, Alolita Sharma and Brandon Harris also provided some support to the Outreach team, by discussing A/B testing requirements with Frank Schulenburg & Lennart Guldbrandsson.
Status: In preparation for the planned deployment of MediaWiki 1.17 on February 8, all outstanding revisions were reviewed. The deployment was attempted twice that day, and eventually postponed because of major performance issues that caused an outage. The problems were investigated, and another plan was published, based on heterogeneous deployment (meaning not all wikis would run the same version of the software). Tim Starling and Roan Kattouw developed wmerrors, a PHP extension to display fatal error pages for PHP. On February 11, a first wave of small wikis were switched to MediaWiki 1.17. On February 16, other small and medium-sized wikis were switched. An attempt to deploy to our biggest wiki (en.wikipedia.org) resulted in a short outage. The English Wikipedia and all remaining wikis were successfully upgraded to MediaWiki 1.17 later that day (read the latest update). Many issues encountered this month were due to the large amount of code changes since the last release (almost 5500 changes reviewed over 7 months). In the future, software deployments should be smaller and happen more regularly, hence reducing the risk of repeated outages.
Status: Now that MediaWiki 1.17 has been deployed to all Wikimedia wikis, remaining bugs are expected to surface and be fixed. We're hoping to release MediaWiki 1.17 soon for third-party users (see draft release notes), but problems related to DBMS support may delay it. Its main feature will be the Resource loader. It will also include category collation improvements. Developers are already discussing MediaWiki 1.18.
Status: Foundation work on this was put on hold pending the 1.17 release. We're now planning on publishing an open request for proposals calling for developers to move this work forward. In the community, Markus Glaser continues to add support for database setup inside the Selenium framework.
Status: We initially attempted to deploy the multicast version of udp2log, but we discovered firmware problems in our routing infrastructure. Our plan is now to have a second machine that receives unicast logging messages that we use for secondary services.
Mobile site rewrite — Port of our existing gateway to another framework for easier support & collaborative development.
Status: We're still in hiring mode looking for a great developer to lead our effors. At the same time, we're also putting together a roadmap for our mobile development, and starting to coordinate research and development. We're drafting a survey now.
Wikipedia version tools — Support and development of a series of tools to select Wikipedia content for offline use.
Status: Currently, offline copies of Wikipedia content are generated by the Wikipedia 1.0 team through use of the release version tools written by User:CBM. Since many in the community would like to see more options, Arthur Richards is actively assessing the codebase on the toolserver to understand the work involved in extending the current toolset.
Status: PediaPress has wrapped up their first development push for adding openZim support to the collections extension. Testers are invited to test the new extension on PediaPress' test wiki. We're now collecting bug reports before deploying it to the live site.
Kiwix UX study — Evaluation of the user experience of the Kiwix mobile app to access offline Wikimedia content.
Status: We've finished our first UX pass over Kiwix and published the recommendations on the Kiwix wiki. Emmanuel Engelhart is implementing some of these new features while we gear up for the next phase of assessment. At the same time, we're engaging with the local Wikimedia community in India to see how well the tool is working.