In October we completed the initial stage of our data center deployment in Dallas; finishing the setup of essential infrastructure, and making sure all essential project data is being replicated and/or backed up in Dallas. The Ops team is now moving towards the next phase; making codfw capable of serving the wikis independently of eqiad, ideally using HHVM.
Tampa data center
On the first day of October we shutdown nearly all remaining servers in Tampa, allowing some room to resolve unexpected problems should they occur. On October 6, we started removing all equipment from their racks, and shipped equipment with remaining value to Dallas. By October 12, we had vacated the facility completely, marking the end of 10+ years of Wikipedia's home in Tampa.
Labs metrics in October:
Number of projects: 145
Number of instances: 413
Amount of RAM in use (in MBs): 1,992,192
Amount of allocated storage (in GBs): 20,395
Number of virtual CPUs in use: 975
Number of users: 4,109
Number of tools: 947
Number of tool maintainers: 550
Labs suffered a brief, partial outage on October 7 due to a presumed out-of-memory failure on a virtualization node. Everything was back up and running after about 30 minutes; Andrew rebalanced instance load to forestall similar such failures.
New hardware has been ordered to expand Labs VM capacity. With luck this will be in place within the next month or so.
Marc redesigned the default base image for labs instances. /var/log is now in its own partition, and we support some limited partition resizing.
In October, the team working on VisualEditor improved some features, prepared the way for table structure editing, and fixed over 60 bugs and tickets.
The keyboard shortcuts for items in the toolbar's menus now show on the right hand side, to make them easier to find. We made a few improvements to the display and editing of templates. Firstly, template fields can now have a value automatically set, like today's date for clean-up notices. Links inside templates now show in red if their targets don't exist, like for the editing surface. Last month's change to warn against saving a template with missing required fields was improved by highlighting them, as is the warning about trying to make an image 0 pixels wide.
The initial tools to edit the structure of tables were finalised and will be made available next month. We made progress on providing a new auto-filled citations tool, and improvements to the link editing and media searching tools, all of which will be coming in the near future.
In October, the Editing team worked on the front-end standardisation project, now one of Wikimedia Engineering's Q2 priorities, as well as the work on VisualEditor, both of which are reported separately. The team welcomed Marielle Volz as a new team member, who will focus on improving citations support. The team created the wikitext concept of <indicator>s, a way to show icons which tell the user something about the page in MediaWiki. These features are now supported in the Vector, Monobook, Cologne Blue and Modern skins, and will replace the need for the local CSS hacks that some wikis have used to approximate this feature. The TemplateData extension now wraps its contents in a <noinclude> tag unless editing a sub-page, and the help link's target can now be set to a page other than the one on MediaWiki. The last vestiges of the CiteThisPage extension were removed from the Cite extension, as part of a complex manoeuvre to simplify the naming of the system; it now has its own home. The team's more significant work, on front-end standardisation, is now reported under that project.
October has seen a lot of maintenance-type work in Parsoid land. We updated our logging infrastructure to send our event logs to LogStash, which makes it far simpler to identify any production errors. We also started a round of code cleanup and improving readability / maintainability to moving to Promises API which eliminates the callback mess common with node.js code. We were involved in debugging some Varnish-related issues that were causing excess timeouts than needed; the issue is now fixed, and we improved the efficiency of queued Parsoid parsing jobs. At the end of October, we also initiated an upgrade of the Parsoid production cluster from node 0.8 to node 0.10.
In non-maintenance work, we continue to fix bugs and compatibility issues with the current default rendering. We've also been working to find entirely CSS-based solutions to citation customizations. This CSS-based approach will simplify customizability, and also free clients from having to worry about MediaWiki site messages. This is still work in progress as we are finding gaps and fixing them.
In October, the French WP Forum des nouveaux offered a Flow board as an alternate way for new contributors to ask for help. The Forum is currently using both a wiki talk page and a Flow board; a goal for the team is to build all of the functionality that the Forum needs in order to switch over completely to a Flow board. The team also built a new modal dialog for moderation actions, put unread Echo notifications in reverse chronology, and worked on the back-end needed for creating a Search feature on a Flow board.
In October, following on from the successful launch of Nearby on iOS in September, the Wikimedia Apps team released Nearby on Android as well. Nearby allows you to see articles about things that are near you, and read their articles to learn more about them. The team also released features that roll up disambiguation and page issues into a button underneath the page title, so that the content of the article is brought closer to the top of the article. We also released a refactor of the technical architecture of the app to bring the app closer to compliance with Google's standards for Android apps, which also brought a few minor UX improvements. On the iOS side, a bug fix build was released which improved the UI scaling on iPads, fixed a few minor issues with the table of contents, and fixed miscellaneous crashes. At the end of the month, a build was submitted to Apple which reworked the way the app retrieves information from the servers, which should increase the stability of the app and reduce crashes.
During October 2014, the Partners engineering team enhanced the partners portal (ZeroPortal) extension to improve security and maintainability, added better font support for the Wikipedia Zero end user experience, migrated the remainder of operator partner configurations to use the unified design in support of lowered cache fragmentation on the mobile web Wikipedias, improved the fidelity of zero-rated Graph extension pageview statistics relative to the legacy Limn graphs while streamlining batch processing, made Wikipedia Zero end user information screen overlays follow MobileFrontend style, created Cucumber tests for the partners portal, added colorized crouton support to the Android app while reducing intrusive warnings on the Android and iOS apps for zero-rating (colorized iOS banner code also merged, will be in forthcoming release), prepared analysis and code for language-aware redirects from the m.wikipedia.org webroot, consulted with partners on API usage and app preloads and HTTP header enrichment and language aware redirects, and examined the interplay of Wikipedia Zero with CentralNotice (e.g., fundraising) banners; currently, CentralNotice is suppressed when Wikipedia Zero page rewriting is in force, but in the future it would be nice to show non-fundraising CentralNotice content.
This month the mobile web team created a new experimental interface for WikiGrok, a feature designed to provide an easy way to contribute structured data to Wikimedia projects. We are currently showing the two different versions of WikiGrok to readers and logged in users on the mobile beta site on English Wikipedia and gathering qualitative and quantitative data on engagement with the feature and quality of contributions. We also focused on backend integration to allow us to generate Wikidata questions on more articles, in preparation for testing at scale on the stable mobile site in the coming months.
CLDR version 26 updates that affected plural rules for MediaWiki i18n have been integrated. The affected messages have been updated and translators have been informed. As part of regular maintenance of the Translate extension, many unused configuration variables have been removed and Special:Translate now works better with custom skins. Niklas also fixed a major regression in a refactoring which broke the message group selector.
The second version was deployed, and Kartik worked with the Operations team and upstream developers of Apertium to prepare requisite packages for the machine translation service. The category adaptation was added, as well as bi-directional machine translation support between Catalan, Portuguese and Spanish. Language support was extended, and development started on the translation dashboard and the 3rd version of Content Translation. The first draft of the graduating language support specification was completed; This specification will guide selection of further language pairs to be supported through the Content Translation tool.
In October we prepared for November in which we deployed Cirrus to all the remaining wikis by installing new servers installing new versions of Elasticsearch and our plugins. We also fixed up regex search which had caused a search outage.
The major engineering development work for single-user login wrapped up this month and there are now three tools being tested on beta labs.
Special:GlobalRenameRequest is the form that users who are renamed post-finalization can fill out to request a new username and submit it to Stewards and global renamers. Special:GlobalRenameQueue allows Stewards and global renamers to manage rename requests in a simplified interface form for post-finalization requests to be sent to from Special:GlobalRenameRequest. Lastly, Special:GlobalUserMerge is in very early testing stages. This tool will allow Stewards to merge the accounts of two or more usernames owned by one person who may have multiple account names due to conflicting usernames.
Once these tools are tested out and deployed, outreach will begin by talk page contact and by email where possible with all accounts that will be renamed due to SUL finalization.
The project kicked off mid-month with the merge of patches that enable PSR-3-based logging by MediaWiki. These changes are being tested in beta and will begin to roll out to the production cluster in early November with the 1.25wmf6 release branch.
Initial work has begun on a Profiler implementation that uses XHProf to collect information about the runtime costs of MediaWiki code. This approach to profiling will enable collection of information on code running on Wikimedia servers without relying on explicit wfProfileIn() and wfProfileOut() calls. This in turn will make splitting code out of MediaWiki core easier.
We completed security reviews for WikiGrok, Labeled Section Transclusion headers, the IEG grant-review application, and RecentActivityFeed. We also released security updates for CentralAuth and MobileFrontend.
This month we welcomed Elena Tonkovidova to the WMF staff as our new QA Tester. Elena will be working closely with Rummana Yasmeen in that role. In support of our QA efforts, we invited Elisabeth Hendrickson, author of the book "Explore It!", to a well-attended meetup at the WMF office for a presentation about Exploratory Testing. Other projects for the month include adding a Ruby style checker Rubocop to our Jenkins builds, creating a Ruby style guide, further enhancements to the Vagrant development environments, updates to the beta labs test environment, and other projects.
This month we began a significant project for improving and refactoring existing browser tests. Starting with the Echo and Flow repositories, we intend to update and improve the browser tests across all the repos to be more modern, flexible, maintainable, and reliable. Our new Ruby style checker Rubocop in all of the Jenkins builds will help greatly in this effort, as will major improvements to the architectures of the features being tested, of which VisualEditor is a notable example.
In October the Wikidata team focused on performance, bugfixing and planning for structured data support for multimedia files. This included a week-long workshop with volunteers and the WMF multimedia team in Berlin. They made progress on statements on properties, simple queries and usage tracking and rolled out first partial changes to the user interface that are part of the new design. They answered questions in an office hour on IRC about structured data for multimedia files and are looking for input on how to improve references and constraints violation reports. And to top it off the project celebrated its second birthday.
The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.