Wikimedia Platform Engineering
From MediaWiki.org
The Wikimedia Foundation Platform Engineering team is responsible for the MediaWiki core, managing work from volunteer developers, as well as for providing services that are used by multiple technology teams. The Platform Engineering group is managed by Rob Lanphier. It is part of the Wikimedia Foundation's engineering department.
See also a series of articles presenting the Platform engineering group:
- What is “Platform Engineering”?
- Data analytics at Wikimedia Foundation
- Technical Liaison; Developer Relations (TL;DR) (former name of the "Engineering community team")
- The MediaWiki Core group
- Meet the Analytics Team
Contents |
MediaWiki Core [edit]
Team [edit]
-
Tim Starling (lead)
-
Munagala "xyvram" Ramanath
Responsibilities [edit]
- Manage the MediaWiki release cycle
- Ensure that MediaWiki core is meeting the evolving needs of the website
- Make quality MediaWiki releases available for others outside of the Wikimedia Foundation
- Develop and document a clear set of APIs so that external developers can create applications that easily interface with MediaWiki
Current activities [edit]
Main projects [edit]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| MediaWiki 1.21 |
[edit]
Deployment of MediaWiki 1.21wmf versions and release of MediaWiki 1.21 |
[edit]
Sam Reed |
Status: (2013-04-monthly)
The 1.21 deployment cycle to Wikimedia wikis is complete, and the MediaWiki 1.21 tarball is being prepared for release, with a target release date of May 15. Mark Hershberger recently released MediaWiki 1.21rc4.
|
| Git conversion |
[edit]
Conversion of the MediaWiki source repository from Subversion to Git |
[edit]
Chad Horohoe |
Status: (2013-04-monthly)
We deployed a first iteration of a Bugzilla integration plugin, which provides notifications to Bugzilla when changes are made in Gerrit. We’ve increased the memory allocated to Gerrit, as well as deployed a couple of other stability fixes; both of these changes should provide some minor performance and stability improvements to users. Finally, we’ve deployed a new version of Gerrit that includes superior garbage collection support. This drastically improved the compression of repositories on-disk, which has resulted in a wide range of improvements for all users for all operations, from cloning to pushing to commenting on changes.
|
| Multimedia |
[edit]
|
[edit]
Rob Lanphier, Michael Dale, Jan Gerber, Aaron Schulz, Faidon Liambotis, Ariel Glenn |
Status: (2013-04-monthly)
Differently-sized video thumbnails now only require one reference thumbnail (for the time position) to be generated. This helps to avoid expensive decoding to derive thumbnails. The Score extension was deployed on April 22nd. It allows users to create and document musical scores on Wikimedia sites.
|
| Wikidata deployment |
[edit]
Deployment of Wikidata related core modifications |
[edit]
Daniel Kinzler, Tim Starling, Sam Reed, Chad Horohoe |
Status: (2013-04-monthly)
After a minor delay due to some job queue and infrastructure migration work, Wikidata Phase II was deployed to all Wikipedia sites. This allows editors to reference and display content from Wikidata inside infoboxes.
|
| Lua scripting |
[edit]
Support for Lua as a scripting language for MediaWiki end-users |
[edit]
Tim Starling, Brad Jorsch (anomie) |
Status: (2013-04-monthly)
Some bugs were fixed and internationalization changes merged this month; no major changes were made. The community continues to develop Lua-based templates, such as the citation templates on the English Wikipedia.
|
| Site performance and architecture |
[edit]
Various initiatives geared toward improving site responsiveness, decreasing resource consumption, and/or improving site maintainability |
[edit]
Tim Starling, Asher Feldman, Ori Livneh, Aaron Schulz, Antoine Musso, Andrew Garrett |
Status: (2013-04-monthly)
All job queues were migrated to JobQueueRedis off of the main DB clusters. Improvements were made to the category update queries to reduce lock exceptions that users often encountered when deleting files. This works via a new transaction callback hook added to the core database class, which can be used to resolve similar problems.
|
| Admin tools development |
[edit]
Feature development and bugfixes on tools for stewards and local power-users |
[edit]
James Forrester (Product Manager), Chris Steipp, Peter Gehres, Tim Starling |
Status: (2013-04-monthly)
This month the team mostly worked on Single User Login finalisation, after which all user accounts will be global across all of Wikimedia's public wikis, allowing for cross-wiki notifications and better tools for editors. This will require all user accounts to be uniquely named and not conflict with other accounts. The global account renaming tool neared initial completion, and the global and local blocking based on XFF was finished and deployed. Work on designing a global CheckUser tool was postponed due to lack of resources.
|
| REST proposal |
[edit]
REST implementation of MediaWiki's web API for greater flexibility, usability, and standardization |
[edit]
Federico Lucignano (Wikia) |
Status: (2013-02-monthly)
Yuri started gathering requirements for the RESTful (content-oriented) API as part of the overall planning for the API v2.0 roadmap. Also, Wikia's focused R&D sprint has led to remove all the remaining obstacles identified previously and Wikia has identified a new, larger dedicated Product team to get to a final implementation of the REST API following the directions set by the existing prototype and internal RFC; the team (API/Data) is the same one in charge of Search and all the related API's, this will ensure a better integration of this product into the new API strategy. At the moment of writing the knowledge transfer required for the team to start this new phase has just begun and the team has to first to complete the current work on another project already in progress. In the meantime, Wikia will make an RFC public for review and feedback.
|
| Search |
[edit]
Search infrastructure maintenance and overhaul |
[edit]
Tim Starling, Munagala Ramanath (Ram), Chad Horohoe |
Status: (2013-05-08)
shool project
|
| Auth systems |
[edit]
Improvement of authentication and authorization systems, and adding new auth mechanisms such as OpenID and OAuth |
[edit]
Chris Steipp, Tim Starling, Brad Jorsch, Aaron Schulz |
Status: (2013-04-monthly)
During April, the team primarily focused on implementing SUL v2, which will fix issues that users are having with new security features in recent browser releases. SUL v2 is ready for testing and deployment is targeted for early May. In addition, the team worked toward a final design specification for OAuth and will begin working on that pending the successful deployment of SUL v2.
|
Continuous responsibilities [edit]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Shell requests |
[edit]
Ongoing processing of Bugzilla requests that require shell access to fix |
[edit]
Sam Reed, |
Status: (2012-10-monthly)
Currently at 92 open (query for latest number), 3414 total resolved since the beginning of tracking (query for latest number), 2924 total fixed (query for latest number), and 38 resolved in October (query for October)
|
| Security auditing and response |
[edit]
Inspection of code and training of developers |
[edit]
Chris Steipp |
Status: (2013-04-monthly)
We released the MediaWiki 1.19.5 and 1.20.4 security releases on April 15th.
|
Planned or backburner projects [edit]
| Project | Description | Project lead & team | Status | |
|---|---|---|---|---|
| API maintenance |
[edit]
Perennial maintenance of the MediaWiki API |
[edit]
Sam Reed |
Status: (2012-12-monthly)
,help
|
|
| Replicated session handling | Create a project page for this activity (how-to) | |||
| HipHop deployment |
[edit]
Support in MediaWiki for the HipHop source code transformer, and deployment to the Wikimedia cluster |
[edit]
Tim Starling, Sam Reed, Chad Horohoe, Mark Bergsma, Domas Mituzas |
Status: (2013-02-monthly)
Corporate business planning and development party New York City
|
|
| App-level monitoring |
[edit]
Implementation of application-level parameters monitoring |
[edit]
Rob Lanphier, Sam Reed |
Status: (2012-08-monthly)
support
|
|
| AcademicAccess |
[edit]
An authentication proxy to allow selected users to access third-party academic publishing sites |
[edit]
Chad Horohoe, Ryan Lane |
Status: (2013-05-03)
Academic Access to Poor People
|
|
| Configuration management |
[edit]
A core feature of MediaWiki to handle a wiki's configuration options through a web interface |
[edit]
Chad Horohoe |
Status: (2011-07-24)
This project was mainly on hold in July.
|
|
| Template profiling | Create a project page for this activity (how-to) | |||
| Continuous external storage recompression | Create a project page for this activity (how-to) | |||
| VipsScaler |
[edit]
Deploy the VipsScaler extension to the Wikimedia cluster |
[edit]
Tim Starling, Bryan Tong Minh |
Status: (2012-01-03)
Deferred until after 1.19 deployment. Bryan Tong Minh will be working with the upstream developers on fixing the problems we found during the test deployment. VIPS will be used for TIFF images after 1.19 without the need for upstream fixes, but PNG and JPG will have to wait for upstream fixes.
|
|
| Database sharding |
[edit]
Enabling sharded tables in MediaWiki |
[edit]
Aaron Schulz, Asher Feldman |
Status: (2012-07-monthly)
Aaron Schulz wrote a new class (ExternalRDBStore) used for sharding tables in MediaWiki, and is now in bugfixing mode. He also wrote a patch to shard some of the tables associated with FlaggedRevs as a first use of this class. Asher Feldman is currently investigating hardware requirements for utilizing sharding.
|
|
| Code review management |
[edit]
Review of changes made to the MediaWiki core code and extensions used on Wikimedia wikis. |
[edit]
Rob Lanphier, Tim Starling, Chad Horohoe, Sam Reed, Aaron Schulz |
Status: (2012-09-monthly)
Volunteers have been of a lot of help in managing the wiki configuration (operations/mediawiki-config) repository. The WMF analytics code review graphs show an uptick in patchsets awaiting review at the end of September, and a Signpost analysis showed (among other statistics) that WMF staff provide 86% of first reviews for core patchsets, and just five staffers collectively account for about 55% of that total.
|
|
| MediaWiki 1.20 |
[edit]
Release of MediaWiki 1.20 |
[edit]
Sam Reed, Mark Hershberger |
Status: (2012-11-monthly)
Mark Hershberger published the MediaWiki 1.20 stable tarball on November 7th. Chris Steipp published a security update (1.20.1) on November 29th.
|
|
Recently completed projects [edit]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Disk-backed object cache |
[edit]
A supplement to memcache-based object cache, which will allow us to greatly increase the amount of space dedicated to caching parsed pages. |
[edit]
Rob Lanphier, Tim Starling |
Status: (2011-07-25)
To improve the MySQL-based version of this system, Domas Mituzas suggested to split the cache into several tables, which Tim Starling implemented in MediaWiki. The system was then deployed on July 11th and the cache has been filling up since then, thus increasing the parser cache hit ratio from about 30% to 80%. Possible future steps include adding previous page revisions to the cache.
|
| Heterogeneous deployment |
[edit]
A system allowing the deployment of different versions of MediaWiki to different production wikis |
[edit]
Rob Lanphier, Tim Starling, Aaron Schulz |
Status: (2012-04-17)
This feature is now deployed on Wikimedia's sites, and we use it to do staged rollouts of new versions of MediaWiki to our wikis.
|
| MediaWiki 1.18 |
[edit]
Deployment and release of MediaWiki 1.18 |
[edit]
Sam Reed, Tim Starling (pretty much everyone else is involved) |
Status: (2011-11-30)
Developers sprinted to fix the last blocker bugs, and the ones uncovered by testers. Sam Reed announced the first beta release and first release candidate of MediaWiki 1.18. Mark Hershberger followed up on comments on the English Wikipedia's Village pump related to the deployment of 1.18, bugs reported in bugzilla, and installation and upgrade reports. Sam announced the final 1.18.0 release on November 27th, as well as the 1.17.1 security release.
|
| MediaWiki 1.19 |
[edit]
Deployment and release of MediaWiki 1.19 |
[edit]
Rob Lanphier |
Status: (2012-05-03)
We have now released the tarball of MediaWiki 1.19. Work on 1.19 is now complete, barring future security updates, and engineers are now working on MediaWiki 1.20.
|
| MediaWiki 1.20 |
[edit]
Release of MediaWiki 1.20 |
[edit]
Sam Reed, Mark Hershberger |
Status: (2012-11-monthly)
Mark Hershberger published the MediaWiki 1.20 stable tarball on November 7th. Chris Steipp published a security update (1.20.1) on November 29th.
|
| Wikivoyage migration |
[edit]
Migrating data and code to set up the new Wikimedia travel site |
[edit]
Erik Moeller, Daniel Zahn, Chris Steipp, Sam Reed, Matthias Mullie |
Status: (2013-01-monthly)
Wikivoyage officially launched on January 15. Most of the Wikimedia Foundation's involvement was completed in November, but some minor bugfixing was done in support of the official launch.
|
Quality Assurance [edit]
Responsibilities [edit]
- Ensure that all new code that is released meets quality standards
- Triage, manage and resolve all bug reports
Team [edit]
- Chris McMahon
- Željko Filipin
Working closely with Michelle Grover in Mobile
Current activities [edit]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Quality Assurance |
[edit]
Software testing and quality assurance for Wikimedia engineering activities |
[edit]
Chris McMahon, TBD |
Status: (2013-05-21)
Tested Visual Editor at Test Summit peer conference, several issues reported. Tested Echo deployments. Upcoming work with Flow, more with GuidedTours, more with ULS.
|
| Beta cluster |
[edit]
Establish a functional, production-like environment in Wikimedia Labs suitable for final-stage testing of new features. |
[edit]
Chris McMahon, Antoine Musso, Sam Reed, Faidon Liambotis |
Status: (2013-05-16)
Ariel Glenn has setup a redis instance on beta (bug 41530). Antoine fixed the jobrunner that was no more processing any jobs because it could not access to the Mediawiki files. Jobs are thus processing again, and from redis!
|
| Continuous integration |
[edit]
A testing platform to provide continuous quality control for MediaWiki |
[edit]
Antoine Musso, Chad Horohoe, Timo Tijhof |
Status: (2013-05-06)
Upgraded Zuul, it now supports project templates in its configuration files and reports back in Gerrit the duration of the job run.
|
| Browser testing |
[edit]
Automated testing of MediaWiki within various browsers to improve quality |
[edit]
Chris McMahon, Željko Filipin, Michelle Grover |
Status: (2013-05-21)
Automated tests continue to find issues, although those issues prevent many builds from being green during maintenance and fixing. Preparing for OPW work to automate tests for VisualEditor, and some ULS in progress also.
|
Analytics [edit]
Team [edit]
Responsibilities [edit]
- Implement and manage Web analytics tool to provide data for other foundation departments and the movement as a whole
- Monitor and analyze key metrics on Wikimedia Foundation websites
Current activities [edit]
| Project | Description | Project lead & team | Status | |
|---|---|---|---|---|
| Analytics infrastructure |
[edit]
A robust, distributed computing and data services platform. |
[edit]
Andrew Otto, David Schoonover, Diederik van Liere |
Status: (2013-04-monthly)
We've improved the functionality of Limn, our visualization tool, to allow users to create and edit charts via the UI. We can also automatically deploy new instances of Limn, so it's faster and easier to setup dashboards. In addition to current users, we expect this to be very helpful for the Program Evaluation team as they start to develop their own analytics.
We're also now importing 1:1000 traffic streams, enabling us to migrate reports from our legacy analytics platform, WikiStats, onto our big data cluster, Kraken. In the future, this will make it easier for us to publish data and visualize reports using our newer infrastructure. We have implemented secure login to the User Metrics API via SSL. We've also introduce a new metric called <code|pages_created, allowing us to count the number of pages created by a specific editor. We improved the accuracy of the udp2log monitoring and upgraded the machines to Ubuntu Precise in order to make the system more robust. |
|
| Analytics Visualization, Reporting & Applications |
[edit]
Limn is a GUI for constructing beautiful visualizations without need of programming skills. |
[edit]
Dan Andreescu, Dave Schoonover |
Status: (2013-04-monthly)
We published our monthly report card. As part of Wikimedia's ongoing mobile initiative, we also helped develop analytics that would support ongoing delivery and planning of mobile functionality:
|
|
| Wikistats |
[edit]
A set of reports providing statistics about Wikimedia sites |
[edit]
Rob Lanphier, Erik Zachte, Diederik van Liere |
Status: (2013-02-monthly)
New mobile pageview report is in testing phase but not ready to ship.
|
|
| Data Releases |
[edit]
Public Dataset Releases |
[edit]
Diederik van Liere, Andrew Otto, Dave Schoonover |
Status: (2013-04-monthly)
We have not introduced any new data sources as part of our analytics platform in April.
|
|
| Analytics/Services & Access Points | Create a project page for this activity (how-to) | |||
| Analytics/Misc | Create a project page for this activity (how-to) | |||
Planned or backburner projects [edit]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Analytics infrastructure |
[edit]
A robust, distributed computing and data services platform. |
[edit]
Andrew Otto, David Schoonover, Diederik van Liere |
Status: (2013-04-monthly)
We've improved the functionality of Limn, our visualization tool, to allow users to create and edit charts via the UI. We can also automatically deploy new instances of Limn, so it's faster and easier to setup dashboards. In addition to current users, we expect this to be very helpful for the Program Evaluation team as they start to develop their own analytics.
We're also now importing 1:1000 traffic streams, enabling us to migrate reports from our legacy analytics platform, WikiStats, onto our big data cluster, Kraken. In the future, this will make it easier for us to publish data and visualize reports using our newer infrastructure. We have implemented secure login to the User Metrics API via SSL. We've also introduce a new metric called <code|pages_created, allowing us to count the number of pages created by a specific editor. We improved the accuracy of the udp2log monitoring and upgraded the machines to Ubuntu Precise in order to make the system more robust. |
Engineering Community Team [edit]
Formerly "TL;DR" (Technical Liaisons & Developer Relations) Old archived notes from weekly meetings (caution: messy)
Team [edit]
-
Sumana Harihareswara (lead)
Responsibilities [edit]
- Attract and support volunteer technologists so that they can become an even more powerful force in driving improvement on the website
- Help nurture volunteer-developed extensions and widgets
- Provide communications support for all groups in the engineering department
- Develop clear documentation for MediaWiki so that new staff, volunteer developers and external users have a smooth ramp-up process to becoming MediaWiki developers
Current activities [edit]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Bug management |
[edit]
Management of Wikimedia's bug tracker. |
[edit]
Andre Klapper |
Status: (2013-05-21)
9
@2 Laj= |
| Mentorship programs |
[edit]
Google Summer of Code, LevelUp, Outreach Program for Women, and other programs that mentor contributors |
[edit]
Quim Gil, Sumana Harihareswara |
Status: (2013-05-21)
We have communicated our pre-selection of candidates to the Google Summer of Code and Outreach Program for Women organizers. More details on May 27, as soon as the public announcements of both programs are made. Quim Gil created a landing page for organizations willing to collaborate with Wikimedia co-organizing technical activities.
|
| Technical communications |
[edit]
An activity aiming to incrementally improve communications between the communities of developers and users |
[edit]
Guillaume Paumier |
Status: (2013-05-03)
Guillaume Paumier focused on assembling, reviewing, editing and publishing the Wikimedia engineering report for April 2013 (and writing its plain English summary), as well as other on-wiki documentation and communications support.
|
| Volunteer coordination and outreach |
[edit]
Coordination of volunteer efforts, technical outreach and recruitment of new volunteers |
[edit]
Sumana Harihareswara, Quim Gil, Guillaume Paumier |
Status: (2013-05-21)
Quim Gil worked on a template for landing page for wikitech-announce that can be used in other Wikimedia mailing lists. Check the discussion and the source code. He also created a landing page for organizations willing to collaborate with Wikimedia co-organizing technical activities.
|
Planned or backburner projects [edit]
Recently completed projects [edit]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| MediaWiki architecture document |
[edit]
An overview of MediaWiki's architecture and decisions that led to it |
[edit]
Guillaume Paumier, Sumana Harihareswara |
Status: (2012-04-17)
The community awaits the publication of Volume 2 of Architecture of Open Source Applications, probably in a matter of days. It will include our now-finished chapter on MediaWiki's architecture.
|
| Wikimedia blog maintenance |
[edit]
Maintenance and new features for the Wikimedia blog |
[edit]
Guillaume Paumier, |
Status: (2012-04-monthly)
Guillaume Paumier made minor changes to the theme, like not displaying trackbacks and pings in the comments list. He investigated plugins to support multiple authors, photo galleries and multilingual posts, and is testing Piwik as a possible solution to record metrics. Chad Horohoe created git repositories for the WP-Victor theme and the WMBlog plugin in Gerrit, and imported their history from GitHub.
|
| Summer of Code 2011 |
[edit]
A sponsored community program allowing students to join the community as developers. |
[edit]
Sumana Harihareswara |
Status: (2012-11-20)
Neil Harris is now working on getting Kevin Brown's ArchiveLinks project completed and deployed. This activity page ("Summer of Code 2011") is now being retired; in the future, status updates will be on the Mentorship programs page.
|
| Summer of Code 2012 |
[edit]
A sponsored community program allowing students to join the community as developers. |
[edit]
Sumana Harihareswara, Gregory Varnum as backup organizational administrator |
Status: (2012-11-20)
The Wikimedia community is following up its Summer of Code participation with participation in the GNOME Outreach Program for Women, which offers paid internships for work in open source software projects. This follows the community's decision not to participate in Google Code-In due to a lack of mentors. Also, Sumana Harihareswara posted a retrospective on our participation in Summer of Code 2012, concluding that we're improving at student retention but need to get better at producing something useful at the end of a mentorship period, and offering suggestions on how to improve our processes in the future. This activity page ("Summer of Code 2012") is now being retired; in the future, status updates will be on the Mentorship programs page.
|
| Wikimedia Foundation engineering 20% policy |
[edit]
A policy and practice whereby WMF engineering staff spent at least 20% of their work time on tasks that directly serve the Wikimedia developer and user community |
[edit]
Sumana Harihareswara, Rob Lanphier, All engineering staff |
Status: (2012-11-monthly)
Sumana Harihareswara has developed Mentorship programs/LevelUp, a new program in which WMF engineers and systems administrators will grow their and the community's capacity for the long term. This replaces 20% time, which is now defunct.
|
