Wikimedia Engineering/Report/2012/October

Engineering metrics in October:
 * unique committers contributed patchsets of code to MediaWiki.
 * The total number of unresolved commits went from about 440 to.
 * About shell requests were processed.
 * About developers got access to Git and Wikimedia Labs.
 * Wikimedia Labs now hosts 137 projects, 694 users; to date 1268 instances have been created.

Major news in October include:
 * a redesign of the mobile site emphasizing readability and navigation;
 * launch of a Wikipedia app for Windows RT and Windows 8 tablets;
 * a test of a redesigned account creation page;

''Note: As of last month, we're proposing a shorter and simpler version of this report for less technically savvy readers.

Upcoming events
Bangalore DevCamp (9–11 November 2012, Bangalore, India)

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

Announcements

 * Željko Filipin joined the Platform engineering team as QA Engineer (announcement).
 * Andre Klapper joined the Platform engineering team as Bug Wrangler (announcement).
 * Michelle Grover joined the Mobile engineering team as a QA contractor (announcement).
 * Luke Welling joined the Features engineering team as Senior Features Engineer (announcement).
 * Brad Jorsch joined the Platform engineering team as Software Engineer, working in the MediaWiki Core group (announcement).
 * Steven Bernardin joined Technical Operations team as Data Center Technician, working in our Tampa data center.
 * Steven Bernardin joined Technical Operations team as Data Center Technician, working in our Tampa data center.

'''Site Infrastructure

 * Mark has successfully implemented range seeking in Varnish as well as fix several video streaming bugs, and finally redeployed Varnish@Eqiad, replacing those upload Squids@Tampa instances. With this success, Mark is now working on replacing those upload Squid instances@ESAMS. He has provisioned several (8) servers and based on his early testing indicated on 1 box,  he could potentially use  less.  Currently we are using 23 squid  (older) servers for upload@ESAMS.


 * In addition, Mark deployed four (4) new Varnish servers to server bits@ESAMS, The existing two(2) are being redeployed for other uses. This will provide higher throughput and added redundancy for the coming Fundraising season.

ULSFO

 * Yes, this is the name of our new West Coast caching center. Earlier this month, Leslie and Daniel went racking and stacking those networking equipment on the 2 racks we have there. Next, we are going to purchase the caching servers once Mark's confirms the configuration.

Data Dumps

 * Compressed multistream format files of current articles are now being produced for all dumps; researchers working with the larger projects may find these helpful. We're working with Amazon on hosting the most recent dumps for EC2 users, thanks to Diederik.  We've encountered some performance issues with media bundle generation off-site and are investigating; at the same we are working on moving that from experimental to production status.

Wikimedia Labs

 * Home directories are being migrated to glusterFS
 * pam_mkhomedir has been enabled; the home directory creation script has been disabled
 * /home is now a direct autofs mount, rather than having indirect mounts for each user under /home
 * Nova, Glance, and Keystone have been upgraded to latest essex stable release
 * Salt has been installed on all instances and puppet has been run on all instances
 * Numerous bugs have been fixed in OpenStackManager (project creation bug, removing tiny flavor from interface) and some features have been added as well (add user to bastion when shell group added)
 * Patch sent into Nova to enforce unique instance names
 * Labs team attended the OpenStack summit in San Diego. One takeaway is that we are leading a team to push a DNS service into OpenStack incubation. Andrew's Nova DNS code will be merged into this project.

Readers
Native Wikipedia application available for Windows 8 - Brion Vibber blogged about his release of an Windows 8 app. This app was not part of the regular Wikimedia product roadmap and instead came out of 1 day an iteration that each team member gets for research time.

Offline
Kiwix

Release 0.9rc2 is almost finished. Highlight of this release is the presence of kiwix-serve for MS/Windows, directly available per click from the Kiwix UI. A first version of kiwix-plug was installed on 15 devices with the project Afripedia. With Wikimedia France, a Wikipedia in French on USB sticks was successfully sold (after one week: no bug dected and no stock anymore) for the first time in France. Work on current projects continues, a few new projects are on the launchpad for 2013 ; more in the next reports.

Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.

The Wikidata team has worked on initial parts of Phase 2 of Wikidata (Infoboxes) and worked together with the WMF to get Wikidata deployed on wikidata.org. A big step towards this was the merge of the content handler branch into MediaWiki core. This allows MediaWiki to now handle more content types than just wikitext. In addition they are looking for help with the initial design of the Main Page of wikidata.org. They also published a draft for how they think propagation of changes from a repository to the clients should work and are looking for feedback.

Feel free to come to one of the next office hours if you have questions. (Or raise them on the mailing list or talk page.)

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.