Wikimedia Engineering/Report/2012/October

Engineering metrics in October:
 * 110 unique committers contributed patchsets of code to MediaWiki.
 * The total number of unresolved commits remained stable around 440.
 * About 35 shell requests were processed.
 * About 57 developers got access to Git and Wikimedia Labs.
 * Wikimedia Labs now hosts 137 projects, 694 users; to date 1268 instances have been created.

Major news in October include:
 * a redesign of the mobile site emphasizing readability and navigation;
 * launch of a Wikipedia app for Windows RT and Windows 8 tablets;
 * a test of a redesigned account creation page;

''Note: As of last month, we're proposing a shorter and simpler version of this report for less technically savvy readers.

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.



Announcements

 * Željko Filipin joined the Platform engineering team as QA Engineer (announcement).
 * Andre Klapper joined the Platform engineering team as Bug Wrangler (announcement).
 * Michelle Grover joined the Mobile engineering team as a QA contractor (announcement).
 * Luke Welling joined the Features engineering team as Senior Features Engineer (announcement).
 * Brad Jorsch joined the Platform engineering team as Software Engineer, working in the MediaWiki Core group (announcement).
 * Steven Bernardin joined the Operations team as Data Center Technician, working in our Tampa data center.

Technical Operations
Site Infrastructure
 * Mark Bergsma has successfully implemented range seeking in Varnish, fixed several video streaming bugs, and finally redeployed Varnish at Eqiad, replacing the upload Squids in Tampa. Mark is now working on replacing upload Squid at Esams. He has provisioned 8 servers, and based on early testing, less may actually be needed. We are currently using 23 older servers for upload Squid at Esams. In addition, Mark deployed 4 new Varnish servers to serve bits from Esams; The existing 2 are being redeployed for other uses. This will provide higher throughput and added redundancy for the coming Fundraising season.

West Coast caching center
 * We've started building out a new caching center in the San Francisco area, called ULSFO. In October, Leslie Carr and Daniel Zahn racked and stacked networking equipment. Next, we are going to purchase the caching servers once Mark Bergsma confirms the configuration.

Data Dumps
 * Compressed multistream format files of current articles are now being produced for all dumps; researchers working with content from the larger wikis may find these helpful. We're working with Amazon on hosting the most recent dumps for EC2 users, thanks to Diederik van Liere. We've encountered some performance issues with media bundle generation off-site and are investigating; we're also working on moving that from experimental to production status.

Wikimedia Labs
 * Home directories are being migrated to glusterFS: pam_mkhomedir has been enabled, the home directory creation script has been disabled, and /home is now a direct autofs mount, rather than having indirect mounts for each user under /home. Nova, Glance, and Keystone have been upgraded to latest Essex stable release. On all instances, Salt has been installed and puppet has been run. Numerous bugs have been fixed in OpenStackManager (project creation bug, removing tiny flavor from interface) and some features have been added as well (add user to bastion when shell group added). A patch was sent into Nova to enforce unique instance names. The Labs team attended the OpenStack summit in San Diego. One takeaway is that we are leading a team to push a DNS service into OpenStack incubation; Andrew Bogott's Nova DNS code will be merged into this project.

Readers
Native Wikipedia application available for Windows 8 - Brion Vibber blogged about his release of an Windows 8 app. This app was not part of the regular Wikimedia product roadmap and instead came out of 1 day an iteration that each team member gets for research time.

Offline
Kiwix
 * Release 0.9rc2 is almost finished. Highlight of this release is the presence of kiwix-serve for MS/Windows, directly available per click from the Kiwix UI. A first version of kiwix-plug was installed on 15 devices with the project Afripedia. With Wikimedia France, a Wikipedia in French on USB sticks was successfully sold (after one week: no bug dected and no stock anymore) for the first time in France. Work on current projects continues, a few new projects are on the launchpad for 2013 ; more in the next reports.

Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.

The Wikidata team has worked on initial parts of Phase 2 of Wikidata (Infoboxes) and worked together with the WMF to get Wikidata deployed on wikidata.org. A big step towards this was the merge of the content handler branch into MediaWiki core. This allows MediaWiki to now handle more content types than just wikitext. In addition they are looking for help with the initial design of the Main Page of wikidata.org. They also published a draft for how they think propagation of changes from a repository to the clients should work and are looking for feedback.

Feel free to come to one of the next office hours if you have questions. (Or raise them on the mailing list or talk page.)

Future
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.