Wikimedia Engineering/Report/2013/July

Engineering metrics in July:
 * 114 unique committers contributed patchsets of code to MediaWiki.
 * The total number of unresolved commits went from around 960 to about 1283.
 * About 40 shell requests were processed.
 * Wikimedia Labs now hosts 168 projects and 1,623 users; to date 2,167 instances have been created.
 * The tools project in Labs now hosts 252 tools and 218 members.

Major news in July include:
 * Giving more editors an easy-to-use editing interface (the VisualEditor) on several Wikipedias
 * Improving language support on our sites via summer interns' projects and easier configuration options, and asking for help translating the VisualEditor interface
 * Enabling users to edit our sites from mobile devices, like phones and tablets, and announcing a future user experience bootcamp focusing on mobile editing
 * Finishing our transition from keeping source code in Subversion to storing it in Git
 * Launching a Wikipedia Zero partnership with Aircel, giving mobile subscribers in India the potential to access Wikipedia at no data cost
 * Updating the Wikimedia movement on how we intend to protect our users' privacy with HTTPS
 * Signing a contract with longtime MediaWiki contributors to manage MediaWiki releases for the open source community
 * Explaining how we find and gather software problems and deliver the fixes to users

''Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.

Upcoming events
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.

For a more complete and up-to-date list, check out the Project:Calendar.

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

Technical Operations
Site infrastructure
 * Lots of Puppet refactoring work got done this month, including considerable reorganization of the puppet masters. Several manifests have been moved into modules, but completing this project will take many months.

Data Dumps
 * The English Wikipedia dumps ran out of our Ashburn data center this month, and so did a number of other big wikis' dumps. There's an issue with the abstract dumps that needs to be sorted out for those, but other than that everything ran smoothly.
 * Petr Onderka has been getting a lot of work done on the incremental dumps. A first preview of the code was announced as well as a proposed binary file format which the program currently uses.  For a preview of what's coming up, you can check the timeline. Your comments and suggestions are welcome!

Wikimedia Labs
 * Though there were some features introduced this month, the majority of our time was spent on documentation, tracking down bugs and improving usability. We had a documentation sprint this month, targeted at improving documentation for the Tool Labs project. Work continued on stabilizing the NFS server -- we believe we've tracked down the stability issues to RAID controller problems. The compute nodes are becoming increasingly low on disk space, but we've tracked this down in a change in behavior of nova and have deployed a fix. nova-network was starting to experience timeouts due to excessive load, leading to instance creation failures. We've extended dhcp renewal times to reduce load. We upgraded wikitech.wikimedia.org and wiktech-static to the 1.22wmf11 version of MediaWiki. We also: deployed the AJAX-enabled delete instance feature; deployed a change to display more informative instance statuses; fixed issues in LdapAuthentication that broke blocking and renaming users; and deployed a change to allow service groups to be added to service groups, to make sharing code and data between tools easier.

Analytics
We reviewed our planning document with the Sue and Erik and the Engineering Directors. Reception was positive and we will be communicating next steps more widely in August. Team's focus on short term deliverables, reliability and hiring in July. We identified 2 potential candidates for Front End/Python work. We have been performing multiple phone screens together with Recruiting. Pipelines are good.

Kiwix
The Kiwix project is funded and executed by Wikimedia CH.

We are preparing the first release of a new Wikipedia ZIM creation solution for August. We also have achieved a new release of Kiwix for Android; this new version includes a few bug fixes and new features. Beside the release of traditional Wikipedia ZIM files, we have also published two interesting ZIM files: one which includes 2.500 ebooks (EPUB & PDF) of French literature and one with the new Wikipedia for Schools selection. ZIM incremental update GSoC project progresses well too: first working versions of zimpatch & zimdiff console tools are available and integration with Kiwix has started. Kiwix developers will be available at Wikimania, during the hacking days and at the WikimediaCH both during Wikimania itself.

Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.


 * In July, we deployed Wikidata to all Wikivoyage sites in all languages, to manage their language links. We updated the continued Roadmap for Wikidata Development. Coveralls.io support has been added to most of our components. Since the first deployment of Phase1 to Wikipedia, about 240 million interwikilinks (5GB text) have been removed from articles (2012 vs 2013 analysis).


 * In other news, the AAAI Feigenbaum Prize for Watson was donated to the Wikimedia Foundation by IBM research to support work, especially on Wikidata.


 * Denny Vrandečić explains why Wikidata items are identified with a Q.

Future

 * The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts. Annual goals for the 2013–2014 fiscal year are currently being drafted.