Wikimedia Engineering/Report/2014/September

Major news in September include:

Engineering metrics in September:
 * 151 unique committers contributed patchsets of code to MediaWiki.
 * About 27 shell requests were processed.

Upcoming events
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.

For a more complete and up-to-date list, check out the Project:Calendar.

Work with us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

* Senior Software Engineer - Services
 * Software Engineer - Services
 * Software Engineer - Maps & Geo - Mobile
 * Software Engineer - Mobile - iOS
 * Release Engineer
 * Technical Writer
 * Full Stack Developer - Analytics
 * Research Analyst
 * Agile Coach/ScrumMaster - Team Practices Group
 * Operations Security Engineer
 * Technical Project Manager
 * UX Senior Designer
 * UX Senior Design Researcher
 * UX User Research Recruiter
 * UX Visual Design Fellowship
 * Mobile Partnerships Regional Manager
 * Project Coordinator - Engineering

Announcements

 * https://lists.wikimedia.org/pipermail/wikimediaannounce-l/2014-September/000987.html
 * https://lists.wikimedia.org/pipermail/wikitech-l/2014-September/078488.html
 * https://lists.wikimedia.org/pipermail/wikitech-l/2014-September/078638.html

Technical Operations
Dallas data center
 * In September we have setup (backup) replication of most project data, including core databases and external storage. Work on Swift images and system backups was still ongoing into October. Essential system infrastructure such as an installation server, DNS, LVS, NTP etc. has been deployed as well.

Tampa data center
 * We started the last push to get the remaining services & systems out of our Tampa data center, with a deadline for shutdown of all systems on October 1st. The remaining services included PDF generation, mail servers, noc.wikimedia.org and LDAP.

Labs metrics in September:
 * Number of projects: 146
 * Number of instances: 415
 * Amount of RAM in use (in MBs): 1,996,288
 * Amount of allocated storage (in GBs): 20,435
 * Number of virtual CPUs in use: 977
 * Number of users: 4,083

 Wikimedia Labs
 * Wikitech (the Labs web interface) is now managed via the standard WMF deployment system. This should allow for more frequent MediaWiki updates and overall greater stability.  Details here.
 * The last historic remaining dependencies on our old Tampa datacenter (e.g. ldap and labs dns backup servers) were finally stamped out and replaced with dependencies on Dallas hardware.
 * One of the labs virtualization hosts (virt1006) was suffering intermittent problems, so all affected instances were migrated to other hosts in order to stave off possible future disaster. Consequently Labs is a bit short on virtualization space, but new hardware procurement is under way.
 * Several long-unused instances and projects were cleaned up in order to free up more space.
 * The last of the ToolLabs replica DB servers was upgraded to MariaDB 10

Kiwix
The Kiwix project is funded and executed by Wikimedia CH.



Work with our partner Bookeen to get an e-ink reader able to read Wikipedia offline has made progress. We have achieved to get a first version of the device firmware (see beside for a screenshot) which will be tested in the field by the Malebooks pilot deployment.

As a consequence of a bug fix session with Parsoid and Wikisource developers at Wikimania, we were finally able to generate usable ZIM files from Wikisource (demonstration here with fr.wikisource.org).

Work on the offline project Gutenberg has continued and we are now almost ready to release. A few ZIM files are in testing, for example here in German or here in Spanish.

Kiwix was represented at the Selenium conference where we have animated a 2 days bug hunting days: 120 bugs were open and 50% were already closed.

Pretty much work were done on our mwoffliner solution to make it easier to use for everyone, in particular to make ZIM files for only a selection of articles. As a demonstration we have prepared a ZIM files with all Wikipedia articles about medicine.

After many years, a new version of a tool to generate a live CD inluding Kiwix and a Wikipedia was released: to be tested.

Wikidata
The Wikidata project is funded and executed by Wikimedia Deutschland.

In September the Wikidata team was focused on improving performance, doing groundwork for the new user interface design and making it possible to track where data from Wikidata is used. Next to that they worked on tests and prepared for a week-long meeting with the WMF multimedia team and volunteers to discuss and plan structured data support for Wikimedia Commons.

Future

 * The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.