Wikimedia Engineering/Report/2011/September

August 2011

Wikimedia engineering report, September 2011

October 2011

Major news in September include:

• The deployment of protocol-relative URLs, paving the way for native HTTPS;
• Major progress by our new Internationalization & localization team;
• The deployment of our new mobile front-end, now the default mobile experience;
• The start of the deployment of MediaWiki 1.18 to Wikimedia sites;
• The completion of the Google Summer of Code project;
• Ongoing preparation for the upcoming coding events in the USA, the UK, India and online;
• A revamped, and now maintained, roadmap for Wikimedia engineering efforts.

Hover your mouse over the green question marks ([?]) to see the description of a particular project.

Upcoming events

• New Orleans hackathon (14–16 October, New Orleans, USA) — Ryan Lane and Sumana Harihareswara continued to plan this coding event, which will focus on tools and infrastructure, and Wikimedia Labs. Discussions about the upcoming migration to git are also planned.
• Weekend of Code (18 October - 8 November, online) — Weekend of code is a planned hacking contest targeting developers worldwide. Awards will be presented for the best gadget, extension, and mobile app. The contest will be advertised via different channels.
• India hackathon (18–20 November, Mumbai, India) — A hackathon will be held concurrently with the WikiConference India 2011, and near its venue. The main focus will be on language, mobile and offline support for MediaWiki content. About 100–150 participants are expected (read more).
• Brighton hackathon (19–20 November, Brighton, England) — Lewis Cawte is preparing a hackathon in Europe dedicated to general MediaWiki hacking.
• Check out the Software deployments page on the wikitech wiki for up-to-date information on the upcoming deployments to Wikimedia sites.

Job openings

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

The following positions are open: Product Manager (Analytics), QA Lead, Director of Features Engineering, Systems Engineer (Data Analytics), Software Developer (Rich Text Editing, Features), Software Developer (Front-end) and Software Developer (Back-end), Product Manager (Mobile), Software Developer (Mobile), Product Manager.

The following requests for proposals are open: Networking Contractor Amsterdam, Internationalization and Localization Feature Development, Logging Analysis, Article Feedback Feature, Development and Operations Engineer, Systems & Operations Engineer, XML Dumps Help.

Operations

Site infrastructure

• Tampa Data Center [?] — Our new data center contractor in Tampa, Chris Johnson, has finalized the installation of three new racks, which will be used for networking, application serving, caching and data storage. Two racks (72 servers) worth of application servers have also been installed. With each server having 12 CPU cores, this is almost doubling our existing application server capacity. To-date, we have deployed and added 58 (of the 72) new application servers to production capacity.

• Virginia Data Center [?] — Ben Hartshorne has completed replicating External storage (article texts) data to our new data center in Ashburn, for disaster recovery and usage by eqiad application servers in the future. Preparations are also being made to deploy bits.wikimedia.org in eqiad using Varnish 3. Jeff Green has been building up the new EQIAD fundraising infrastructure and securing those boundary servers.

• Media Storage [?] — Russ Nelson has continued work on the SwiftMedia extension for support of Swift as a MediaWiki foreign media file repository, and image thumbnail scaling. The integration of Swift with MediaWiki will also be one of the topics of the upcoming Hackathon in New Orleans.

• HTTPS — Protocol-relative URLs have been enabled on all sites by Roan Kattouw. SSL termination servers have arrived and are in the process of being installed. HTTPS should be fully enabled in early October.

• Lowlights — There were three short outages on September 26th, 2011.

Testing environment

• Virtualization test cluster [?] — All services except for DNS are up. The puppet repository has been released in a public repository. We've switched to using git/gerrit for our production puppet process. New instances have been tested building from scratch using puppet. We've started giving accounts out to some early testers of Labs via labsconsole.

Backups and data archives

• Data Dumps [?] — The September English Wikipedia run completed in under 8 days. The dump of all text revisions is now available in a number of smaller files rather than several giant ones. The dump system has been converted to work with heterogeneous deployment and has undergone some testing with the upcoming MediaWiki 1.18 codebase.

Features Engineering

Editing tools

• Visual editor — Trevor Parscal expanded the software design documentation. He also refactored some of the data structures to follow a model/view controller pattern and support document-level transactions. He wrote tests for the new structure, and got it to render paragraphs and lists; he's now working on rendering tables.
• Internationalization and localization tools — Siebrand Mazeland created a list of initial Indic target languages (and their properties) to improve support in MediaWiki for India's 28 official languages. Roan Kattouw deployed the Babel extension and found issues in the Translate extension, which were fixed by Niklas Laxström. Niklas started to rewrite MediaWiki's logging system to better support internationalization. He also added real-time statistics on translation completion, fixed a number of i18n bugs and reviewed code. Santhosh Thottingal implemented interface changes in the Narayam extension (suggested by Trevor Parscal), which was deployed to select Wikimedia wikis. He fixed bugs and added features to the WebFonts extension, and researched new possible fonts and their license. Alolita Sharma continued to plan the India Hackathon with Siebrand, Santhosh, and the local India team. Last, the i18n/l10n team met in person in San Francisco in late September.

Participation and editor retention

• Article feedback — Dario Taraborelli continued to analyze the data recorded. Howie Fung assessed proposals from vendors in response to the RfP to build an extended review system, that would notably integrate free-text comments to the Article feedback feature.
• LiquidThreads 3.0 — Andrew Garrett worked on the integration between the LiquidThreads interface and the new back-end. What remains now is mostly to re-implement administrative actions, logging integration, new messages, and other ancillary parts of LiquidThreads on top of the new back-end. Once this is done, there will be a new, more stable version available for use on Wikimedia wikis.
• MoodBar — Timo Tijhof fixed bugs, notably with Internet Explorer. Howie Fung analyzed the data recorded; the users' most frequent complaints and questions concerned page creation, file upload, the text editor, and having their edits reverted.
• Feedback Dashboard — Brandon Harris expanded the scope of this dashboard to support different sources, and updated the design. Roan Kattouw started to implement the feature, aided by Trevor Parscal for the HTML and CSS part.
• Article creation and patrol — In response to a request by a majority of English Wikipedia community members to restrict new page creation to autoconfirmed users, discussions are underway to improve the article creation workflow and the user interface for new page patrolling as alternative strategies to cope with the new page creation backlog and reduce high-friction interactions with new users.

Multimedia Tools

• UploadWizard — Ian Baker, Neil Kandalgaonkar and Jeroen De Dauw fixed a number of bugs, notably related to the Wiki Loves Monuments campaign and the deployment of protocol-relative URLs to Wikimedia Commons. Ian also researched solutions for multi-file selection and AJAX uploading. Jeroen and Neil worked on a feature to support custom licenses.

MediaWiki infrastructure

• ResourceLoader — Roan Kattouw completed the back-end for .js/.css page search suggestions, overhauled the Gadgets API, reviewed code, and greatly improved caching. Timo Tijhof continued to work on the gadget manager and the AJAX gadget editor, which is nearing completion: displaying, modifying, saving of gadgets and autocompletion in all form fields is now implemented. Roan and Timo also prepared a plan for a prototype setup to test shared gadgets in a wiki farm environment.

Wikimedia Labs

• Multimedia — Neil Kandalgaonkar and Ian Baker reviewed some of Michael Dale's code. Alolita Sharma and Michael discussed a timeline for testing, code review and deployment, as well as related hardware requirements.

Mobile

• Mobile Research [?] — Mani Pande and Parul Vora continued to work on the report on their field research in India and Brazil. They completed phone interviews in San Francisco, Chicago and Dallas (with partner firm Answerlab) and started to schedule in-person interviews.

• MobileFrontend — MobileFrontend was deployed in September and is now the default Wikipedia mobile experience. We've reached out to our various communities to create custom main pages, and are rolling out new ways of viewing Wikimedia projects on mobile. We also migrated our old WAP gateway traffic to MobileFrontend, so that we can serve our users from one place. Next we'll be iterating on our roadmap to guide our work.

Special projects

Fundraising support

• 2011 Fundraiser — Ryan Kaldari added increased logging for CentralNotice changes, including interfaces and filters to search and review them. Katie Horn abstracted and refactored the DonationInterface extension in preparation for supporting a new potential payment processor, GlobalCollect. The team also fixed a number of bugs and added new features (see the retrospective of sprints 5 & 6). Jeff Green continued general work to improve the resiliency and reliability of the fundraising architecture.

Offline

• Kiwix UX initiative [?] — The Kiwix team built out a beta portal for feedback this month, where users can now easily submit their thoughts on using Kiwix. Next, we'll be adding some in-app feedback to offer multiple ways of sending ideas and feedback.

Platform Engineering

MediaWiki Core

• MediaWiki 1.18 — Wikimedia engineers worked together relentlessly in September to ready MediaWiki 1.18 for deployment. They finalized the review of the code, and fixed all the issues they could find. The deployment to Wikimedia sites was split into several phases using the heterogeneous deployment system. Stages 1 and 2 were completed on select wikis without too much trouble. Deployment to all remaining Wikimedia sites is scheduled for October 4th, 2011.
• Code review management — Even though engineering and code review efforts were focused on MediaWiki 1.18 in September, the backlog of unreviewed commits in trunk still continued to decrease (see chart), which means we will be able to release MediaWiki 1.19 fairly rapidly, possibly as soon as December 2011. Work on MediaWiki 1.19 is scheduled to start as soon as MediaWiki 1.18 is officially released.
• API maintenance — Sam Reed continued to maintain the API and worked on integrating the new Wikimedia dashboard with external data sources.
• Shell requests — Sam Reed continued to go through the backlog of shell requests and to process them. Because of the number of new requests, the backlog has remained around 60 open requests.
• Continuous integration — Chad Horohoe worked with Daniel Zahn to set up the dedicated server in our Virginia data center. Its configuration was automated with Puppet. TestSwarm remains to be pupettized. The server is expected to be put in production in early October.
• Wikitext scripting — Tim Starling did some prototyping and testing on this project. He created a test harness that can be used to trial the various approaches, and made a Wikitext-to-Lua template converter to test the relative performance.
• Git conversion — Rob Lanphier proposed a draft plan to proceed to a migration to git over the following months, and got a positive response from developers.

Wikimedia analytics

• Wikimedia Report Card 2.0 — Erik Zachte fixed bugs in Wikistats and continued to automate the process of statistics generation. He also started to publish summaries for all Wikimedia wikis, using the India report card as a model. Nimish Gautam continued to work on improved mobile logs analysis, general data quality issues, and integration of targets into the dashboard.

Technical Liaison; Developer Relations

• Bug management — Mark Hershberger continued to hold thematic bug triage sessions; in September, those focused on UploadWizard, Internationalization and Wikibooks & Wikisource. A calendar of upcoming triages is now available. He also reached out to people directly by e-mail to remind them of FIXMEs that were assigned to them. An upcoming focus on his work will be the patches submitted via Bugzilla.
• Summer of Code 2011 — All seven GSoC students passed. Most projects are in a good shape, even if not totally complete. Sumana Harihareswara is now working on getting the students' code reviewed and polished before it can be merged and used on production wikis.
• Engineering project documentation — Guillaume Paumier continued to update project documentation pages and to write engineering reports.
• Volunteer coordination and outreach — Sumana Harihareswara reached out to the OCaml and Lucene communities to collaborate on math code and search respectively, and coordinated the testing of 1.18 by volunteers. She also worked on organizing upcoming coding events in New Orleans, India, Brighton and online.

Added 30 Nov: in September 2011, 6 developers received commit access, all of whom were volunteers.

• MediaWiki architecture document — A handful of developers contributed to the project. Sumana Harihareswara reached out to developers to convince them to participate, while Guillaume Paumier continued to go through historical content to document the evolution of MediaWiki's architecture.