Wikimedia Engineering/Report/2011/June

Major news this month include:
 * Visual editor, -1 to 100
 * community testing on mobile front-end
 * community testing for kiwix download manager
 * 1.17 Release
 * starting of Google Summer of Code
 * code review
 * code review

Recent events

 * 2011 Wikimedia fundraising summit (June 17-19, Vienna, Austria) —
 * Open Source Bridge conference (June 21-25, Portland, Oregon, USA) — Sumana Harihareswara presented new features in MediaWiki 1.17 and gathered offers of volunteer help (especially around database support, testing, bug triage, and right-to-left support). She also gave a talk about technology management, and recruited candidates for the Wikimedia Foundation's current job openings (read more).

Upcoming events

 * OSCON (July 25-29, Portland, Oregon, USA) — A delegation of about a dozen Wikimedia engineers will be attending the Open Source Convention in July. Many of them will give talks (see full schedule).
 * Wikimania (August 2-7, Haifa, Israel) — Another delegation of about a dozen Wikimedia engineers will be attending the Wikimania conference in August. Besides the Developer Days and wm2011:OpenZIM Developers Meeting, they will also give several talks; the full schedule is now available.
 * Check out the Software deployments page on the wikitech wiki for up-to-date information on the upcoming deployments to Wikimedia sites.

Job openings
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles. The following positions have opened this month: RfPs: The following positions are still open: In addition, we hope to post the following positions over the next few months:
 * Product Manager — Features
 * Product Manager — Analytics
 * QA Lead
 * Operations Engineer — Networking
 * Director of Features Engineering
 * Internationalization and Localization Outreach
 * Internationalization and Localization Feature Development
 * Software Developer — Features
 * Systems Engineer — Data Analytics (previously Data Analytics Engineer)
 * Operations Engineer
 * Senior QA Engineer
 * Networking Contractor — Amsterdam
 * Software Developer, Rich Text Editing — Features
 * Product Manager — Features
 * Software Developer Front-end
 * Software Developer Back-end
 * Release Engineer
 * Technical Writer

Short news

 * CTO leaving WMF — Danese Cooper announced in early June that she would be leaving as Chief Technology Officer of the Wikimedia Foundation at the end of July.
 * Promotions and role changes — Erik Möller announced a reorganization of the engineering management team, promoting Rob Lanphier to Director of Platform Engineering, Tomasz Finc to Director of Mobile and Special projects, Tim Starling to Lead Platform Architect and Mark Bergsma to Lead Operations Architect. Alolita Sharma is serving as acting Director for Features Engineering, and Erik Möller assumed the position of VP of Engineering and Product Development (read more).
 * Other changes — Guillaume Paumier officially transitioned from Product Manager to the position of Technical Communications Manager.

Site operations
Virginia Data Center — Installation of a world-class primary data center for Wikimedia Foundation websites. equinix build up network set up for EQIAD (Equinix Ashburn) - (http://torrus.wikimedia.org/torrus/Network?path=%2FExternal_links%2Feqiad%2FTransit%2FInOut_bps ) IP Transition to Ashburn now available as well connectivity between equinix & tampa (http://torrus.wikimedia.org/torrus/Network?path=%2FInternal_links%2FTransport_pmtpa_eqiad%2FInOut_bps) puppet master server installed and put into service at EQIAD as well about 20 DB server have been provisioned, another 30 are in the works started replicating thumb server from ms5 (tampa) to ms1004 (eqiad). we'll start replicating DBs next week next: other servers
 * Status:

Media Storage — Improvement of our media storage architecture to accommodate expected increase in media uploads. Russ sent an update ; CT to forward it to Guillaume - done
 * Status:

Testing environment
Virtualization test cluster — Environment to deploy temporary machines for testing and experimentation, for use by WMF staff and volunteers working on important projects (as capacity allows). Ask Ryan slowing down in favor of HTTPS work
 * Status:

Backups and data archives
Data Dumps — Improvement of processes to create and provide public copies of public Wikimedia data. started to puppetize things in snapshot2 & snapshot4 starting on snapshot3 running in parallel, as many threads as possible and see failed dumps when trying 32 threads ask Ariel for more info; looking for the bottleneck (CPU? memory? etc.)
 * Status:

Other activities
waiting for the network connectivity; now it's up (as of mid-June) backups will start in July
 * Backups —
 * Asher & Ryan created the systems infrastructure for the Summer Research Associates to perform data mining and analysis work.


 * HTTPS & IPv6 — Ryan Lane announced that Wikimedia sites would be switching to protocol-relative URLs in July, as part of the work to properly support HTTPS.


 * June 23 outage — Production wikis suffered from a 45-minute outage on June 23; a postmortem was drafted, as well as a more general incident response document.
 * Server decommission donations — Rob Halsell announced that decommissioned Wikimedia servers would be donated to non-profits, and called for applicants. Applications are now closed.

Editing tools
Visual editor 0.1 — Exploratory work to identify & prototype initial ideas for a visual editor for MediaWiki. FlaggedRevs — A feature to allow changes made by logged-out and new users to be reviewed before they appear as the primary version of an article.
 * Status: Trevor Parscal continued to work on the front-end of the visual editor, and specifications for accessing the editing surface via the API. A hybrid rendering approach appears to be the best strategy for the visual editor. Neil Kandalgaonkar continued to work on the middleware, DOM and transactions. Neil also continued to work on a demo to integrate MediaWiki and Etherpad. With Alolita Sharma, they planned their upcoming sprints.
 * Status: Aaron Schulz improved user preferences and changed the way statistics are stored in the database, among other minor improvements .Chad Horohoe helped review the backlog of unreviewed commits.
 * Program manager: Alolita Sharma

Content Quality and Editorial Tools
Article Feedback — A feature to collaboratively assess article quality and incorporate reader ratings on Wikipedia.
 * Status: Additional features were added in June. An additional dashboard tracking articles receiving low ratings was added. In response to community feedback, tooltips were added to provide more information on the meaning of the star ratings. Roan Kattouw is currently implementing the UDP back-end to provide clicktracking metrics to assess user engagement. The community provided feedback and bug reports, and the development team addressed the concerns raised, for example by implementing a user preference to hide the tool. Dario Taraborelli continued to evaluate the data provided by the articles already showing the feature. The incremental roll-out to all articles on the English Wikipedia is planned to be completed by mid-July.

Discussions and Interactions
WikiLove 1.0 — An extension to encourage expressions of appreciation between users. MoodBar 0.1 — A feature to encourage new users to provide feedback. rename a section to have a -1 to 100 header StructuredProfile -- A set of features that a) make it easier for new editors to fill out their profile pages with meaningful information about their background and interests and b) surface select profile information to experienced editors within lists such as recent changes, watchlist, etc. The ideas are still in development, so please provide feedback on the StructuredProfile Talk page.
 * Status: Jan Paul Posma completed the back-end work and fixed bugs. Roan Kattouw reviewed the code and enabled the extension on a private production wiki for testing. The feature was then enabled on a public prototype wiki that was used for informal user testing. The extension is planned to be deployed to production wikis on June 30.
 * Status: Brandon Harris updated the feature's design, while Andrew Garrett and Timo Tijhof have completed a first iteration of development on both front-end and back-end (see code in SVN).

Multimedia Tools
Upload wizard — A feature that provides an easier way of uploading files to Wikimedia Commons, the media library associated with Wikipedia.
 * Status: Neil Kandalgaonkar continued to fix bugs, and added an additional functionality to show thumbnails before upload in modern browsers.

Other projects

 * Non-Roman character set localization — Alolita Sharma published two RfPs to assemble a dedicated team to tackle localization issues from a feature development and outreach perspective. She is currently reviewing applications and contracts.
 * ResourceLoader 2.0 — This project was mostly on hold in June due to the lack of engineering resources. Work is planned to resume in July.
 * LiquidThreads 3.0 — WMF work on this project was mostly on hold in June due to limited resources, that were affected in priority to supporting the 2011 Board Election and the MoodBar.

Wikimedia Labs
Media projects — A set of features to improve media handling and key infrastructure support tools, many developed with Kaltura, such as Metavid, MwEmbed, and the Video Editor. iterating with brion on getting code review updating & fixing his code based on the comments and working on a test plan for TMH for user testing to run on prototype, then deploy later
 * Status:

Parser — Groundwork for the next generation visual editor of MediaWiki. invited the dev community to use the parserplayground and provide feedback
 * Status: Brion Vibber continued to work on the parser plan, and also moved the "parser playground" gadget to an extension (read more).

Mobile
Mobile Research — A research project to help determine our Mobile strategy. Mobile site rewrite — Port of our Ruby-based mobile gateway to PHP.
 * Status: We completed our fieldwork in Brazil in June, consisting of 16 interviews, led by Parul Vora and Mani Pande in three cities in Brazil: Sao Paulo, Salvador and Porto Alegre. We conducted extensive in-home interviews with three kinds of participants in Brazil: readers of Wikipedia on the mobile phone, potential mobile readers (users who read Wikipedia on their desktop, but have the potential of reading it on their phone) and editors of primarily Portuguese Wikipedia, and to  a lesser degree English Wikipedia. We also received about 6 proposals from research firms in the US to conduct research in potentially three cities in the US. We are currently reviewing the proposals and will select the research firm soon. The mobile survey is scheduled to be launched in end of July.
 * Status: Tomasz Finc sent a call for testers to help test the prototype in English, Japanese and Hebrew. Feedback is now being addressed. The mobile team has been actively taking this feedback and adding bugs for easy fixes and new feature requests. As our community has been testing both Patrick and Asher have been actively working together to profile the MobileFrontend extension to prep it for deployment. Were looking at how to integrate it with our Varnish and Squid caching architecture so that we can have the advantages of the WURFL mobile device database but not a sigignifcant performance hit. When its ready we'll do an opt in beta to continue getting everyones feedback before we turn it on as the default experience.

Special projects
Data Competition: We announced the Wikipedia Participation Challenge, a data modeling competition where contestants are tasked with developing an algorithm to predict the number of edits a Wikipedia editor makes. The contest runs from July 28 to September 20, 2011. An anonymous donor has donated a total of USD$10,000 in prize money, which will be split among four prizes.

Fundraising support
2011 Fundraiser — Support and development for the annual fundraiser of the Foundation.
 * Status: Arthur Richards, Ryan Kaldari and Katie Horn are wrapping up the first official development sprint for the 2011 Fundraiser and preparing for the next one beginning on July 5. They are focussing on new features for CentralNotice to better facilitate banner management as well as back-end enhancements to the donation processing pipeline.

Offline
Wikipedia version tools — Support and development of a series of tools to select Wikipedia content for offline use.
 * Status: GSoC student Yuvi Panda is making excellent progress porting the WP 1.0 Bot to a MediaWiki extension. With mentorship from Arthur Richards and support from WP 1.0 Bot author/maintainer User:CBM, Yuvi has implemented an assessment template processing feature and is currently working on a WP 1.0 bot replacement feature to automatically include real-time assessment statistics on project pages.  He'll begin implementing a feature to allow article filtering and selection based on assessment criteria sometime in July.

Collections — General work for Collections extension. Kiwix — Improvement of the user experience of the Kiwix app to access offline Wikimedia content.
 * Status: Tomasz Finc had active discussions between PediaPress, WMF IT, and others to focus our next set of work. We're looking at potentially scaling the collections extension to output much larger collections, integrating it with the new Kiwix download manager, improving general user experience and making it a general purpose solution for article selection.
 * Status: We blog posted the findings from our Berlin Usability study and released the next beta release of Kiwix. Users of Kiwix can now easily download new openZim files right within the interface. No longer will users have to scour the web to find out where to find openZim files. It's all integrated together now. In the future we're looking at connecting it to the collections extension so that anyone can easily download new books collections. If you're interested in helping please check out our volunteer program. Were especially in need of someone who can help out with some complicated libtool issues. Let us know if you can help.

Mobile issues (error 500)

 * Asher Feldman upgraded the Mobile ruby gateway to a newer version of Ruby and Passenger which cleared up a lot of produciton issues. We're no longer seeing error code 500's for image resources and we've cut our service times dramatically.

MediaWiki Core
MediaWiki 1.18 — The upcoming MediaWiki release. merge 1.18 and code review? Code review management — Review of changes made to the MediaWiki code.
 * Status: Tim Starling announced the release of MediaWiki 1.17.0. Thanks to the efforts of the code review team, the backlog of unreviewed commits for MediaWiki 1.18 was drastically reduced in June (see chart). Mark Hershberger started a discussion about which extensions to bundle with MediaWiki 1.18.
 * Status: The backlog of unreviewed commits continued to decrease in June. A long but productive discussion between developers happened on the wikitech-l list about how to further improve the code review process. It led to a proposal of a "20% policy", according to which every eligible Wikimedia engineer would spend 20% of their time doing service work that directly benefits the rest of the community.

Wikimedia analytics
Wikimedia Report Card 2.0 — Usability improvements and streamlining of the creation of the monthly report card. MetricsReporting extension: http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/MetricsReporting/
 * Status: Erik Zachte and Nimish Gautam started a development sprint and worked on the back-end infrastructure, supported by Asher Feldman & Sam Reed. The information stored in a database is accessed via the MediaWiki extension API, and the visualization part uses JQplot. The team hopes to demonstrate a prototype for the next report card in early July.

Technical Liaison; Developer Relations
Bugmeistering — Management of our bug tracker. Summer of Code 2011 — A sponsored community program allowing students to join the community as developers. Engineering project documentation — An activity to ensure that project documentation of Wikimedia engineering activities is complete and up-to-date. volunteer recruitement?
 * Status: Mark Hershberger continued conducting bug triages to surface issues that require attention or decisions; this month these meetings switched from phone to IRC to improve transparency and accessibility to the community. After helping developers wrap up the 1.17 tarball, he started looking at 1.18 bugs, and led a triage to narrow down the list of open bugs blocking deployment of MediaWiki 1.18 on Wikimedia Foundation sites.  He also worked on several concerns raised by the community, such as enabling "International" numerals on Hindi wiki with Priyanka's help, and right-to-left and extension bundling issues.  He also did internal knowledge sharing on his Bugzilla API client.
 * Status: Our eight Summer of Code students continued working on their projects full-time, and all are now committing code. WMF staffers and community members are mentoring the students, assisted by Sumana Harihareswara. They are preparing for their mid-term evaluations, to take place in mid-July.
 * Status: Guillaume Paumier finalized the infrastructure for project pages, using templates and transclusion. Because of the tools' limits, full automation wasn't possible. He also continued to update project pages and statuses. A sprint is planned in July to catch up with the documentation of all projects.

Other activities

 * API maintenance — Sam Reed continued to fix bugs and to add new features to the MediaWiki API.
 * Shell requests — Priyanka Dhanda continued to process shell requests, after it appeared that a harmonization of wiki configuration options wouldn't be a significant time saver.
 * Access to Subversion — Volunteer development coordinator Sumana Harihareswara is now the primary point of contact for commit access requests. About 7 new developers were granted commit access in June, among which were 2 Summer of Code students, and 2 Wikimedia Foundation employees.
 * Heterogeneous deployment — Priyanka Dhanda and Tim Starling added features and improved the code, which is now in SVN. Priyanka used it to deploy different versions of MediaWiki on a prototype. Tim and Priyanka are now discussing edge cases and remaining tasks.
 * translatewiki.net support — The list of 500 most used MediaWiki interface messages was updated to help translators focus on the messages with the most impact. The Translate extension may be reviewed in July to be used for content translation on Wikimedia sites, e.g. on meta-wiki.
 * Academic publications authentication proxy — Chad Horohoe started a project whose goal is to allow selected Wikimedians to access third-party academic publishing sites to help with verifiability. The authentication challenges this project entails are not trivial; Ryan Lane is also involved in this project, particularly with his previous experience with OpenID (which maybe used as a mechanism for tying to the CentralAuth). (remove occurrences of "project")
 * HipHop support — HipHop support is still planned to be part of MediaWiki 1.20. In the meantime, we're looking for volunteers to help us package it for different distributions. Please contact Sumana Harihareswara if you have experience with packaging or would like to get involved in this area.
 * Disk-backed object cache — A deployment was attempted on June 27, but had to be rolled back. Tim Starling and Domas Mituzas are investigating what went wrong, and another deployment attempt will be scheduled soon.
 * Projects on hold — The App-level monitoring and Configuration management projects were delayed in June, in favor of other work like the 1.17 release. Some of them will be resumed in July.