Major news in October include:
Engineering metrics in October:
- 150 unique committers contributed patchsets of code to MediaWiki.
- About 27 shell requests were processed.
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.
For a more complete and up-to-date list, check out the Project:Calendar.
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.
- Marcel Ruiz Forns joined the Analytics Development team (announcement).
- Guillaume Paumier joined the Product and Strategy team as Senior Analyst (announcement).
- Bartosz Dziewoński joined Wikimedia as a Features Contractor (announcement).
- Moriel Schottlender converted to full time staff member status as a Software Engineer in Features Engineering (announcement).
- Marielle Volz joined the Wikimedia Foundation as a Software Engineer for the Editing team (announcement).
Dallas data center
- In October we completed the initial stage of our data center deployment in Dallas; finishing the setup of essential infrastructure, and making sure all essential project data is being replicated and/or backed up in Dallas. The Ops team is now moving towards the next phase; making codfw capable of serving the wikis independently of eqiad, ideally using HHVM.
Tampa data center
- On the first day of October we shutdown nearly all remaining servers in Tampa, allowing some room to resolve unexpected problems should they occur. On October 6, we started removing all equipment from their racks, and shipped equipment with remaining value to Dallas. By October 12, we had vacated the facility completely, marking the end of 10+ years of Wikipedia's home in Tampa.
Labs metrics in October:
- Number of projects: 145
- Number of instances: 413
- Amount of RAM in use (in MBs): 1,992,192
- Amount of allocated storage (in GBs): 20,395
- Number of virtual CPUs in use: 975
- Number of users: 4,109
- Number of tools: 947
- Number of tool maintainers: 550
- Labs suffered a brief, partial outage on October 7 due to a presumed out-of-memory failure on a virtualization node. Everything was back up and running after about 30 minutes; Andrew rebalanced instance load to forestall similar such failures.
- New hardware has been ordered to expand Labs VM capacity. With luck this will be in place within the next month or so.
- Marc redesigned the default base image for labs instances. /var/log is now in its own partition, and we support some limited partition resizing.
- Yuvi has made a good start on setting up instance state monitoring.
- Marc added an Ubuntu Trusty bastion and exec node to Tool Labs. Some work is still in progress here due to package renames and dependency changes.
- Marc wrote many pages of operator documentation for Tool Labs.
Editor retention: Editing tools
In October, the team working on VisualEditor improved some features, prepared the way for table structure editing, and fixed over 60 bugs and tickets.
The keyboard shortcuts for items in the toolbar's menus now show on the right hand side, to make them easier to find. We made a few improvements to the display and editing of templates. Firstly, template fields can now have a value automatically set, like today's date for clean-up notices. Links inside templates now show in red if their targets don't exist, like for the editing surface. Last month's change to warn against saving a template with missing required fields was improved by highlighting them, as is the warning about trying to make an image 0 pixels wide.
The initial tools to edit the structure of tables were finalised and will be made available next month. We made progress on providing a new auto-filled citations tool, and improvements to the link editing and media searching tools, all of which will be coming in the near future.
The deployed version of the code was updated five times in the regular release cycle (1.25-wmf3
In October, the Editing team worked on the front-end standardisation project
, now one of Wikimedia Engineering's Q2 priorities
, as well as the work on VisualEditor, both of which are reported separately. The team welcomed Marielle Volz
as a new team member, who will focus on improving citations support. The team created the wikitext concept of
s, a way to show icons which tell the user something about the page
in MediaWiki. These features are now supported in the Vector, Monobook, Cologne Blue and Modern skins, and will replace the need for the local CSS hacks that some wikis have used to approximate this feature. The TemplateData extension now wraps its contents in a
tag unless editing a sub-page, and the help link's target can now be set to a page other than the one on MediaWiki. The last vestiges of the CiteThisPage extension were removed from the Cite extension, as part of a complex manoeuvre to simplify the naming of the system; it now has its own home
. The team's more significant work, on front-end standardisation
, is now reported under that project.
October has seen a lot of maintenance-type work in Parsoid land. We updated our logging infrastructure to send our event logs to LogStash, which makes it far simpler to identify any production errors. We also started a round of code cleanup and improving readability / maintainability to moving to Promises API which eliminates the callback mess common with node.js code. We were involved in debugging some Varnish-related issues that were causing excess timeouts than needed; the issue is now fixed, and we improved the efficiency of queued Parsoid parsing jobs. At the end of October, we also initiated an upgrade of the Parsoid production cluster from node 0.8 to node 0.10.
In non-maintenance work, we continue to fix bugs and compatibility issues with the current default rendering. We've also been working to find entirely CSS-based solutions to citation customizations. This CSS-based approach will simplify customizability, and also free clients from having to worry about MediaWiki site messages. This is still work in progress as we are finding gaps and fixing them.
In October, the French WP Forum des nouveaux offered a Flow board as an alternate way for new contributors to ask for help. The Forum is currently using both a wiki talk page and a Flow board; a goal for the team is to build all of the functionality that the Forum needs in order to switch over completely to a Flow board. The team also built a new modal dialog for moderation actions, put unread Echo notifications in reverse chronology, and worked on the back-end needed for creating a Search feature on a Flow board.
Front-end libraries standardization
In October, following on from the successful launch of Nearby on iOS in September, the Wikimedia Apps team released Nearby on Android as well. Nearby allows you to see articles about things that are near you, and read their articles to learn more about them. The team also released features that roll up disambiguation and page issues into a button underneath the page title, so that the content of the article is brought closer to the top of the article. We also released a refactor of the technical architecture of the app to bring the app closer to compliance with Google's standards for Android apps, which also brought a few minor UX improvements. On the iOS side, a bug fix build was released which improved the UI scaling on iPads, fixed a few minor issues with the table of contents, and fixed miscellaneous crashes. At the end of the month, a build was submitted to Apple which reworked the way the app retrieves information from the servers, which should increase the stability of the app and reduce crashes.
During October 2014, the Partners engineering team enhanced the partners portal (ZeroPortal) extension to improve security and maintainability, added better font support for the Wikipedia Zero end user experience, migrated the remainder of operator partner configurations to use the unified design in support of lowered cache fragmentation on the mobile web Wikipedias, improved the fidelity of zero-rated Graph extension pageview statistics relative to the legacy Limn graphs while streamlining batch processing, made Wikipedia Zero end user information screen overlays follow MobileFrontend style, created Cucumber tests for the partners portal, added colorized crouton support to the Android app while reducing intrusive warnings on the Android and iOS apps for zero-rating (colorized iOS banner code also merged, will be in forthcoming release), prepared analysis and code for language-aware redirects from the m.wikipedia.org webroot, consulted with partners on API usage and app preloads and HTTP header enrichment and language aware redirects, and examined the interplay of Wikipedia Zero with CentralNotice (e.g., fundraising) banners; currently, CentralNotice is suppressed when Wikipedia Zero page rewriting is in force, but in the future it would be nice to show non-fundraising CentralNotice content.
Mobile web projects
This month the mobile web team created a new experimental interface for WikiGrok, a feature designed to provide an easy way to contribute structured data to Wikimedia projects. We are currently showing the two different versions of WikiGrok to readers and logged in users on the mobile beta site on English Wikipedia and gathering qualitative and quantitative data on engagement with the feature and quality of contributions. We also focused on backend integration to allow us to generate Wikidata questions on more articles, in preparation for testing at scale on the stable mobile site in the coming months.
CLDR version 26 updates that affected plural rules for MediaWiki i18n have been integrated. The affected messages have been updated and translators have been informed
. As part of regular maintenance of the Translate extension, many unused configuration variables have been removed and Special:Translate now works better with custom skins. Niklas also fixed a major regression in a refactoring which broke the message group selector.
The second version was deployed
, and Kartik worked with the Operations team and upstream developers of Apertium to prepare requisite packages for the machine translation service. The category adaptation was added, as well as bi-directional machine translation support between Catalan, Portuguese and Spanish. Language support was extended, and development started on the translation dashboard and the 3rd version of Content Translation. The first draft of the graduating language support specification
was completed; This specification will guide selection of further language pairs to be supported through the Content Translation tool.
In October we prepared for November in which we deployed Cirrus to all the remaining wikis by installing new servers installing new versions of Elasticsearch and our plugins. We also fixed up regex search which had caused a search outage.
The major engineering development work for single-user login
wrapped up this month and there are now three tools being tested on beta labs.
Special:GlobalRenameRequest is the form that users who are renamed post-finalization can fill out to request a new username and submit it to Stewards and global renamers. Special:GlobalRenameQueue allows Stewards and global renamers to manage rename requests in a simplified interface form for post-finalization requests to be sent to from Special:GlobalRenameRequest. Lastly, Special:GlobalUserMerge is in very early testing stages. This tool will allow Stewards to merge the accounts of two or more usernames owned by one person who may have multiple account names due to conflicting usernames.
Once these tools are tested out and deployed, outreach will begin by talk page contact and by email where possible with all accounts that will be renamed due to SUL finalization.
Library infrastructure for MediaWiki
The project kicked off mid-month with the merge of patches that enable PSR-3
-based logging by MediaWiki. These changes are being tested in beta
and will begin to roll out to the production cluster in early November with the 1.25wmf6 release branch.
Initial work has begun on a Profiler
implementation that uses XHProf
to collect information about the runtime costs of MediaWiki code. This approach to profiling will enable collection of information on code running on Wikimedia servers
without relying on explicit wfProfileIn() and wfProfileOut() calls. This in turn will make splitting code out of MediaWiki core easier.
Security auditing and response
We completed security reviews for WikiGrok, Labeled Section Transclusion headers, the IEG grant-review application, and RecentActivityFeed. We also released security updates for CentralAuth and MobileFrontend.
This month we welcomed Elena Tonkovidova to the WMF staff as our new QA Tester. Elena will be working closely with Rummana Yasmeen in that role. In support of our QA efforts, we invited Elisabeth Hendrickson, author of the book "Explore It!", to a well-attended meetup at the WMF office for a presentation about Exploratory Testing. Other projects for the month include adding a Ruby style checker Rubocop to our Jenkins builds, creating a Ruby style guide, further enhancements to the Vagrant development environments, updates to the beta labs test environment, and other projects.
Quality Assurance/Browser testing
This month we began a significant project for improving and refactoring existing browser tests. Starting with the Echo and Flow repositories, we intend to update and improve the browser tests across all the repos to be more modern, flexible, maintainable, and reliable. Our new Ruby style checker Rubocop in all of the Jenkins builds will help greatly in this effort, as will major improvements to the architectures of the features being tested, of which VisualEditor is a notable example.
Multimedia Quarterly Review Slides
In October 2014, the multimedia team started work on the Structured Data project, co-hosting a weeklong bootcamp in Berlin with community members and the Wikidata team. Participants developed first ideas for data models and user interface designs, as well as a high-end API prototype to read and write machine-readable data on Wikimedia Commons. These first ideas are now being documented here on Commons and in these project slides, for discussion purposes. If you would like to get involved, we invite you to sign-up for the newsletter and join the discussions on the Structured data hub. You can also learn more about the related metadata cleanup drive in this blog post. Our next office hours chat on Structured Data will take place on Thursday, November 20 at 19:30 UTC, on
#wikimedia-officeconnect (Freenode IRC).
The team also developed and released more improvements to Media Viewer, based on feedback from the recent community consultation and user research. These improvements include: an easier way to disable Media Viewer for personal use, as well as re-enable it from a file page, pre-render thumbnails on backend for new uploads and make MediaViewer text larger in Monobook. Our last improvement for this release will be a caption or description right below the image, to be launched in November. In parallel, we also continued our code refactoring for the UploadWizard, and are now analyzing metrics for the upload funnel analysis, to gradually fix the most critical bugs that cause error messages during upload.
Lastly, the team hosted a quarterly project review
for multimedia and planned its activities for this quarter, as shown in these slides
. For more information about our work, join the multimedia mailing list
A very active bugday on Offline content generation / PDF rendering
took place, organized and led by User:Nemo_bis
. Wikimedia Bugzilla received several security updates (related RT tickets 1
). A new component for IEG was created
. Andre cleaned up some older Backport flags and removed the "Mysterious future" Target Milestone in the Wikimedia product
Wikimedia announced its participation in the 9th round of the FOSS Outreach Program for Women
. We started the selection process
with 15 applications received. We started preparing Wikimedia's proposal for Google Code-in
. We participated in the Google Summer of Code Reunion
, a special summit celebrating 10 years of this pioneering program.
Volunteer coordination and outreach
Analytics/Research and Data
This month we started recruiting
a full-time research analyst dedicated to traffic and readership research and analytics.
We created a number of reports on mobile readership, including: daily snapshots of pageviews by device class and access method; a report on OS version breakdown for iOS app users and overall app usage; updated stats on the impact of the tablet redirect.
We started the documentation on mobile microcontributions and provided instrumentation requirements and experimental design support for a series of tests planned by the Mobile team to be rolled out in this quarter. We also started background research on missing claims in Wikidata as a function of item classes and language, to help inform the design of new types of microcontributions.
We hosted our monthly research showcase, with a presentation by Aaron Halfaker on Wikipedia as a socio-technical system and a guest talk by BarcelonaMedia's David Laniado on sentiment analysis of editor discussions.
We made substantial progress on defining reader engagement metrics. We also supported several external requests for data, including a request to extract a complete set of references with PubMed Identifiers and a dump of revert metadata for the English Wikipedia.
We completed a preliminary analysis of HHVM's effect on newly registered editors
The Kiwix project is funded and executed by Wikimedia CH.
- We released a minor version (1.91) of Kiwix for Android fixing a few annoying issues. Preparation to welcome Google Code-in junior developers has started.
The Wikidata project is funded and executed by Wikimedia Deutschland.
- In October the Wikidata team focused on performance, bugfixing and planning for structured data support for multimedia files. This included a week-long workshop with volunteers and the WMF multimedia team in Berlin. They made progress on statements on properties, simple queries and usage tracking and rolled out first partial changes to the user interface that are part of the new design. They answered questions in an office hour on IRC about structured data for multimedia files and are looking for input on how to improve references and constraints violation reports. And to top it off the project celebrated its second birthday.
- The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.