Engineering metrics in October:
- 134 unique committers contributed patchsets of code to MediaWiki.
- The total number of unresolved commits went from around 1020 to about 1122.
- About 35 shell requests were processed.
Major news in October include:
Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.
For a more complete and up-to-date list, check out the Project:Calendar.
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.
- Ori Livneh transitioned to the Platform Engineering group as Senior Performance Engineer (announcement).
- Gergő Tisza joined the Platfom Engineering group as Software Engineer on the Multimedia team (announcement).
- Leslie Carr and Ryan Lane were both promoted to the position of Senior Operations Engineer (announcement).
- Rummana Yasmeen joined the Platfom Engineering group as Software Test Engineer on the QA team, working primarily on VisualEditor (announcement).
- Vibha Bamba was promoted to the position of Senior User Experience Designer (announcement).
- The team continued to heavily refactor the Puppet configuration: manifests for MySQL, nginx, SSH, puppetmaster, and several others are now properly organized as modules. Alexandros Kosiaris has made considerable progress towards supporting multiple puppetmasters on our cluster, which will greatly improve Puppet performance.
- Andrew Bogott has been working with Yuvaraj Pandian to get the new proxy system properly deployed; this will greatly reduce our need to hand out public IPs to labs users. We're close to hiring a contractor to help with the upcoming migration of Labs from Tampa to Ashburn.
Editor retention: Editing tools
In October, the VisualEditor team continued to improve the stability and performance of the system, and add new features. The deployed version of the code was updated five times (1.22-wmf20
). Beyond fixing bugs, the focus of the team's work this month was to make a number of large structural changes to make the system more dependable and extensible, and continue to make some usability improvements. For example, you now need to press the "delete" key twice to delete a template, reference or image; the first time, they only become selected, to avoid accidental deletion of infoboxes and similar content. A new feature, empowering users to switch from editing in VisualEditor to editing wikitext directly without having to save the page, was also implemented.
In October, the Parsoid team continued to refine the parser behavior in edge cases. Performance was improved by increasing the parallelism of API requests and separating page updates from dependency-triggered updates in the job queue. The round-trip testing server
performance was improved so that we can now run round-trip tests on 160,000 pages over night. Support for private wikis was also added this month.
We also made additional progress on Rashomon
, the revision storage service based on Apache Cassandra. Rashomon is initially going to be used for implementing HTML and metadata storage for Parsoid output. Rashomon was deployed on a test cluster and import/write tests were performed
In October, we released Notifications worldwide
on about 800 different sites, including most of the remaining Wikipedia wikis and 'sister projects'. Fabrice Florin and Keegan Peterzell managed the community outreach for these final releases, while Benny Situ proceeded with the technical deployments. Community members we've spoken to generally find this tool helpful, across languages and world regions. Notifications are also available on mobile devices, and seem to be adding value on these platforms as well. We now plan a final release of Notifications on the German and Italian editions of Wikipedia at the end of November. To learn more, visit our project hub
, read the help page
and join the discussion on the talk page
Flow Portal/Project information
In October, the Flow team implemented a new visual design treatment
on the Flow prototype
(currently hosted on WMF Labs), and we continued to work through the set of features needed for a minimum viable product (MVP) on Wikipedia. We held an in-office User Experience workshop
, primarily with users new to Wikipedia, to get feedback on the usability of the new design. We're planning to demo the Flow MVP to interested WikiProjects in November to get more feedback on what's needed for a first on-Wikipedia trial.
In October, the Growth team completed its sixth and final major A/B test of the GettingStarted
extensions for the onboarding new Wikipedians
project. Data analysis results
for this test were also completed and published, making way for rolling out the winning version on non-English Wikipedias in November. This month, the team also completed background research and early designs for its upcoming work on anonymous editor acquisition
and Wikipedia article creation
2013 Wikimedia fundraiser
The fundraising team has been steadily ramping up to the big English fundraiser, and so the fundraising tech team has been transitioning into their usual year-end fundraising monitoring and support role. At the end of October, we went in to a feature freeze for CentralNotice, and have been concentrating on bugfixes with the intent of increasing stability on the payments cluster and in the rest of the donation pipeline.
We have also started planning for the international fundraising campaigns targeted for early 2014.
Wikipedia Education Program
This month, we fixed several bugs in the Education Program extension and added one much-needed UI feature. More improvements are in the pipeline. For now, we're focusing on issues that are not linked to the extension's lack of integration with ContentHandler, and on features that might carry over to a new version of the extension. We've also begun the groundwork for redesigning the UX and broadening the extension's scope.
Many changes were made to the iOS Commons app in October. Users can now select the license among the most common ones, and swipe to delete categories on yet-to-be-uploaded images. A number of visual and interface improvements have been made, including on the settings page and the image details slider, notably for iOS 7 compatibility. Icon consistency has been improved throughout the app. The app now also has better internationalization support (via Autolayout), as well as better landscape and iPad support.
We started redirecting ineligible access attempts against subdomains zero.wikipedia.org to a unified warning page; this will reduce residual content in search engine caches, and reduce charges for users accidentally visiting zero.wikipedia.org subdomains while off participating networks.
We added support for partner self-management of configuration for JSON-oriented users; this will be enhanced with stage gating before any actual push to production.
Last, we nearly finalized IP address zero-rating lists (with the support of the Operations team), and we've also started working on Wikipedia Zero support for all Wikimedia projects.
Mobile web projects
The mobile web team has been focusing on a variety of new features, as well as running tests and gathering metrics on new and new-ish mobile editors. The team has also been working on general design enhancements to improve the user experience and make the look and feel consistent across all aspects of the mobile experience.
Feature highlights from the past month include: a better AbuseFilter support for mobile editing, early experimentation around integrating VisualEditor with MobileFrontend for tablets, client-side performance enhancements, the ability to 'thank' from the watchlist, and Captcha support for mobile editing.
In beta, 'Near this page' is now available, as well as user profiles and 'Keep Going' (a series of calls to action for new editors).
- The Language Engineering team worked on building a new Main Page for translatewiki.net, the primary translation platform for MediaWiki projects. This includes a new sign-in screen and project displays. A special feature is also being introduced for new translators through which they can provide a limited set of translations for evaluation.
- As part of the enhancements towards Universal Language Selector (ULS), a new font has been developed for displaying Language Names in their own scripts. This is known as the Autonym font and contains only the characters needed for the language names. Significantly small in size, this font will soon be deployed on Wikimedia projects. The team also worked on fixing a font inheritance issue for the edit area (bug 53734).
- The monthly release of the MediaWiki Language Extension Bundle (MLEB) was also completed.
- The team prepared for the open-source language summit to be held in Pune, India on November 18−19 in collaboration with Red Hat and other open-source internationalization developers.
The last deployment from the 1.22 MediaWiki release was on October 24. To see all the changes that occurred during the 1.22 development cycle, see the main 1.22 release
page or the RELEASE NOTES
file. Final release of MediaWiki 1.22 is expected to be November 29
Site performance and architecture
We implemented logging, aggregation & graphing of the VisualEditor DOM load & save timing. We also rolled out
mw.inspect, a library for inspecting static asset metrics. We configured stable URLs and improved cache headers for font resources, and rolled out a localStorage module caching to test wikis and the beta cluster.
Admin tools development
This activity is still officially on hold. However, some development of the global rename user tool is under way at gerrit:92468
In October, CirrusSearch was deployed as a secondary search engine to Wikidata, all Wikivoyage wikis, and Wikipedia in Bengali. It became the primary search engine on Wiktionary in Italian, Wikipedia in Catalan and Wikisource in English. In November, we plan to deploy many more wikis including some larger than the Catalan Wikipedia. To expand to those larger wikis, we've negotiated some new hardware that should be deployed mid month.
We continued to refine the OAuth UX with the design team, and completed all major development tasks for the initial OAuth product. The first third-party application approved to use OAuth, "Gerrit Patch Uploader", was successfully used by several end users. We plan to finish the OAuth deployment in November.
Wikimania Scholarship app
Security auditing and response
We responded to several issues reported in core and extensions. An emergency password reset was put into place to address a private data security issue
In October the Quality Assurance project brought on a new software tester for VisualEditor, Rummana Yasmeen, who became productive quickly, reporting a number of issues and validating a VE deployment. QA spent a significant amount of time with the WMF Language team, using automated browser tests to find issues with the UniversalLanguageSelector and other Language software. QA continues to expand the utility of the beta labs test environment, supporting the new Flow extension there, and increasing the role of beta for deployments. Membership in the QA mailing list continues to grow, and October saw more contributions to the automated tests from volunteers. QA will participate in the Google Code In project as well.
October has been dedicated to consolidating the Jenkins configuration to make it easier to edit. Most actions are now handled by shell scripts under
; editing the scripts doesn't require updating Jenkins jobs. The second slave server has been added to production and is successfully running PHPUnit tests. The packaging of dependencies required to upgrade Zuul has been completed, and Antoine Musso now has a version working in Labs. Finally, we investigated the possibility of running the browser tests whenever a change is submitted in Gerrit; that work is still in progress.
Thanks to Carl Fürstenberg
's work during the Summer, we are now able to build some Debian packages straight into Jenkins using a dedicated instance and the Jenkins Debian glue scripts
. The jobs are listed in Jenkins
under the Ops-DebGlue view.
Quality Assurance/Browser testing
October saw a major change to the architecture of the browser tests. Since they are now in many different repositories (VisualEditor, UniversalLanguageSelector, MobileFrontend, Flow, etc.), we consolidated the code shared among the various tests in all the repositories into a single library. This allows all the tests, present and future, to use the best and most up-to-date infrastructure. As for the tests themselves, we expanded coverage and identified issues for VisualEditor, Flow, gadgets, Language features, MobileFrontend, and more.
Bugzilla now offers a new guided bug entry form
which will make creating good bug reports easier for newcomers. Bugzilla now also displays metadata changes
of a bug report inline for all logged-in users, so they can see in the comments who changed a value of a field (without clicking on "History"). Daniel Zahn upgraded Wikimedia Bugzilla to latest version 4.2.7
. Legoktm mass-imported about 400 Pywikibot tickets
from Sourceforge to Bugzilla. On a related note, Amir ran a PyWikibot Bug Triage
resulting in nearly 100 tickets receiving updates. Furthermore, Andre Klapper investigated Wikimedia Bugzilla's customizations in CSS and code in order to clean up and sync with the upstream code base, to simplify current maintenance and also make a potential future upgrade of Wikimedia Bugzilla from version 4.2 to 4.4 easier.
We presented the summary of GSoC 2013 and FOSS OPW round 6
at the Wikimedia Metrics Meeting. The document about lessons learned in mentorship programs
was updated. Wikimedia applied to Google Code-In
and was successfully accepted on November 1. We opened the call for candidates for FOSS Outreach Program for Women - Round 7
Besides ongoing communications support for the engineering staff, Guillaume Paumier
published three issues of the Tech newsletter
in October, and started a list of open tasks
related to technical communications, available for anyone to work on, primarily in preparation for Google Code-In
Volunteer coordination and outreach
Our proposal for a Wiki devroom at FOSDEM
was accepted and we launched the Call for participation
. On tech community metrics
, we obtained first results of the Community metrics: Who contributes code
key progress indicator. We facilitated the MediaWiki 1.22 Release Plan tech talk
and the Flow UX Workshop
meet-up. The list of participants of the Architecture Summit 2014
was published. We also experimented with a monthly cycle of Engineering Community Team meetings
In October, we continued to expand our multimedia team
and hired Gergő Tisza
as software engineer. Mark Holmquist worked with Gergő to develop a first beta version of the Media Viewer
, which displays images in larger size or full screen; this work was based on designs by Pau Giner, May Tee-Galloway and Jared Zimmerman.
We also completed development on the Beta Features program, which invites users to try out new features before they are released for everyone. A first version of both products is now ready for testing by logged in users on MediaWiki.org (to try new features, click on the small 'Beta' link next to your 'Preferences').
We plan to release the Beta Features program in coming days to Wikimedia Commons and Meta-Wiki, then to all wikis worldwide at the end of November. Fabrice Florin managed the development of both projects, and updated our multimedia plans to prepare for roundtable discussions with community members next month.
Bryan Davis started work on improving the thumbnail pipeline and guided the development of an upcoming GLAM Toolset for batch uploads by museum curators. To discuss these features and keep up with our work, we invite you to join the new multimedia mailing list
. We are also recruiting for a senior software engineer
position on our team.
The team designed and implemented multiple data center configuration support for Kafka (message bus). A bug involving buffer space allocation was exposed and fixed in the Varnish module, and infrastructure work done on automated data ingestion and partitioning. Product Analyst Oliver Keyes did some exploratory work with Hive and provided feedback to the Development team on ease of use and use cases.
The Threshold metric was implemented, and numerous bugs were fixed.
We created dashboards for several Wikipedia Zero partners (Orange Madagascar, Bangalink, Umniah Jordan), and identified and fixed Wikipedia Zero data issues in collaboration with the Zero team.
Analytics/Research and Data
This month, we continued to support Growth
as the team's focus areas for this quarter. We published the results
of the latest GettingStarted
test run by the Growth team, we completed the cohort analysis
for Mobile user acquisition and we worked with the Mobile team to prepare the launch a new test for new user activation
, currently underway.
We analyzed active editor trends to determine whether the Wikipedia total active editors data for September 2013 (~67k) represented an anomalous change from seasonality and the long-term trend, and concluded that this was not the case. The results of this analysis call for the need to apply time series analysis and forecasting methods to other key performance indicators that the Foundation publishes on a daily or monthly basis.
We continued to work with the analytics engineers to provide requirements for Wikimetrics (with a particular focus on UserMetrics feature parity), and to perform data QA and validate the output of the application for metrics that were recently implemented.
We completed a round of consultations with internal stakeholders to identify research needs of each team in the organization and determine their priority. We presented a review of our activities for Q1 and plans for Q2 at the Analytics Quarterly meeting. We identified "metric standardization" as one of the goals the team will focus on in this quarter.
the inaugural Wiki Research Hackathon, a global event hosted in 8 locations in 5 countries, bringing together Wikimedia researchers, academics and community members to work on wiki research projects. The hackathon — the first event organized in the context of the Labs2
initiative — will take place on November 9, 2013.
The Kiwix project is funded and executed by Wikimedia CH.
- We have released a new version (1.4) of Kiwix for Android; besides other improvements, we have introduced a localized user interface. We continue our efforts to speed-up the ZIM throughput; Wikipedia is now available offline in more than 60 languages. The last flash drives from our Deaddrops & Geocaching pilot were sent; we hope to get them walled up before the end of the year. We have also submitted a few proposals about Kiwix for Android for the Google Code-In program.
Slides from the Wikidata presentation at the metrics and activities meeting for October
The Wikidata project is funded and executed by Wikimedia Deutschland.
- At the beginning of the month, Lydia Pintscher took over the product management for Wikidata. In a blog entry, she elaborated her goals for the future. The developers continued their work on the numbers datatype, sorting of statements and simple queries. In addition, a prototype for a multilingual picture dictionary was published, as well as a tool to visualize time and location data from Wikidata and another one to build lists based on data in Wikidata.
- October was also the month to celebrate Wikidata's first birthday. Enjoy some essays, interviews and more from the community.
- The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.