Wikimedia Engineering/Report/2014/December
Major news in December include:
- the third version of the Content translation tool;
- a look at collaboration practices within the mobile web engineering team;
- lead images prominently featured in the Beta version of the Wikipedia for Android app;
- the migration of the Technical Operations team from RT to Phabricator;
- a retrospective on how the deployment of HHVM improved editing performance on Wikimedia sites.
Upcoming events[edit]
There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.
For a more complete and up-to-date list, check out the Project:Calendar.
| Date | Type | Event | Contact |
|---|---|---|---|
| 9 January 2015 | 
|
Tech Talk: API client libraries 1830-1900 UTC in #wikimedia-office connect. | rfarrand |
Personnel[edit]
Work with us[edit]
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.
- Director of Engineering
- Senior Software Engineer - Services
- Software Engineer - Flow (Front-end)
- Software Engineer - Mobile - Android
- Application Security Engineer
- Full Stack Developer - Analytics
- Agile Coach
- Scrum Master
- Senior Technical Product Manager
- Community Liaison
- UX Senior Designer
- UX Senior Design Researcher
- UX Visual Design Fellowship
Announcements[edit]
- Nirzar Pangarkar joined the User Experience team as User Experience Designer (announcement).
Technical Operations[edit]
Labs metrics in December:
- Number of projects: 155
- Number of instances: 437
- Amount of RAM in use (in MBs): 2,147,840
- Amount of allocated storage (in GBs): 21,745
- Number of virtual CPUs in use: 1,059
- Number of users: 4,599
Tool metrics:
- Number of tools: 1002
- Number of tool maintainers: 559
Wikimedia Labs
- We've added two new large servers to the Labs virtualization cluster in eqiad. This will increases our capacity by about 40%; Andrew is now in the process of rebalancing load among servers and allocating new resources to projects that need them.
- Yuvi has made some progress with an intermittent DNS issue. With luck this will be more stable now.
- The toolserver is now officially decommissioned.
Editor retention: Editing tools[edit]
In December, the team working on VisualEditor introduced a new design for the front-end system that VisualEditor uses, improved several existing features, and fixed over 80 tasks, bugs and requests.
The new design of the interface is based on the future path set by the Wikimedia User Experience team, and is intended to gel with designs across Wikimedia's tools and services and the wider direction of travel for the Web as a whole. The interface is subject to revision and improvements, and nothing is set in stone; feedback is appreciated from users and designers alike.
We also made a number of improvements to language editing and browser stability, progress on providing a new auto-filled citations tool, and improvements to the link editing and media searching tools, all of which will be coming in the near future.
The deployed version of the code was updated three times in the regular release cycle (1.25-wmf11, 1.25-wmf12 and 1.25-wmf13).In December, we finished work supporting templates that generate attributes of a table as well as content of the table and do not fit well within the DOM-based model that Parsoid works with. Besides that, we improved error reporting when handling images that lets clients like Flow and VE handle them better. With a view to reducing the HTML size that needs to be loaded and parsed by clients, we stripped the private data-parsoid attribute from templated content since it is unnecessary. We also continued with code cleanup and pay back technical debt. Specifically, we did a bunch of fixes in our nowiki handling when HTML is serialized to wikitext. We improved robustness, correctness, and reduced the number of scenarios where nowikis are needed for quotes. We also made it simpler to detect nowiki scenarios for other wikitext constructs and specifically applied it to links of all flavors.
Collaboration[edit]
- UniversalLanguageSelector:
- Made the width of the ULS panel configurable according to the number of languages.
- Removed the map. The analytics showed that very few people click it.
- Improve the visual design of the Compact Language Links beta feature (done by Niharika Kohli, design and review by Pau Giner and Santhosh Thottingal)
- Improve the performance of language search (done by Thiemo Mättig, review by Santhosh Thottingal)
- Add ability to rely on default configuration when loading messages in jquery.i18n.
- Positioning fixes for the Input Method selector.
- Translate
- Multiple fixes to translation memory
- Refactored and redesigned the article and language selector using ULS. (This required changes in the design of ULS itself.)
- Multiple fixes in section alignment in the translation interface.
- Warning about possible overwriting of the translated page (same title, same topic)
- Show license text on entry points.
- Disabled all ContentTranslation features unless the user enabled the beta feature.
- Automatic draft translation saving (one of the most requested features from user feedback sessions).
- Multiple other minor bug fixes.
- Preparation for deployment in January - configuration, puppetization, etc.
MediaWiki Core[edit]
Library infrastructure for MediaWiki
Documentation on the addition of external libraries to MediaWiki core is underway. An RFC on guildelines for extracting, publishing and managing libraries has also been drafted and will be under discussion by the MediaWiki developer community. A blog post describing the accomplishments and next steps for the continuation of the project is being drafted as well and is expected to be published in January 2015.
Antoine Musso and Timo Tijhof are finalizing a standard practice for testing Composer managed projects using CDB as a test bed for refining the techniques.
Structured logging implementation is continuing with the use of Monolog and Redis transport being tested in the beta cluster and a subset of the production Wikimedia Foundation cluster. A new Monolog handler was written and upstreamed to Monolog to support the use case of randomly sampled log event streams. Chris Steipp also contributed a security related patch to Monolog based on a security review he did for deployment on the Wikimedia Foundation cluster.
The project's "top priority" status expires with the end of the fiscal year quarter, but organic contributions and a raised awareness of the functionality offered by library extraction and integration of high quality third party code is expected to continue. A list of future projects has been started and Bryan Davis and others who have participated in the project have high hopes for the transformative changes that this project has unblocked in MediaWiki.Security auditing and response
Multimedia[edit]
In December 2014, the team fully migrated its tasks and workflow to Phabricator.
Active maintenance was performed on Media Viewer and CommonsMetadata, which have both seen several minor improvements. The long tail of small bugs affecting Media Viewer in particular was a source of several Google Code-in tasks, which several volunteers tackled.
UploadWizard has been the continued focus of major code cleanup and test coverage increase efforts. Bugs were fixed along the way as well.
Thumbnail chaining was deployed for a short period of time and subsequently undeployed following concerns about over-sharpening it caused. Investigations regarding the performance improvements or lack thereof introduced by thumbnail prerendering at uploaded time and thumbnail chaining were conducted. The discussion and further research are happening on our mailing list.
Significant work has gone into creating an extension to support Sentry, a tool designed to log javascript errors. This effort should benefit the entire organization and give engineering visibility on the live client-side errors encountered by users. In our team's case, it is being worked on in order to identify client-side errors that might be happening in UploadWizard and which wouldn't have been the subject of bug reports yet.
For more information about our work, join the multimedia mailing list.
Engineering Community Team[edit]
- Neta Livneh: Wikipedia article translation metrics weekly reports.
- Roxana Necula: Wikipedia article translation metrics weekly reports.
- Priyanka Jayaswal: Pywikibot: Compat to core migration weekly reports.
- Anke Nowottne: Wikipedia Education Program need-finding research weekly reports.
- Ankita Shukla: Collaborative spelling dictionary building tool weekly reports.
- Manpreet Kaur: Extending PyWikiBot support to sites on IWM weekly reports.
Christy Okpo's project "Improving the Wikimedia Performance Portal" had a delayed start due to a last minute problem finding a mentor available. She finally started working in a metrics project for Parsoid (weekly reports).
Wikimedia's participation in Google Code-in 2014 during December evolved as planned, with the participation of several students and mentors.Volunteer coordination and outreach
We kicked off a collaboration with a research team at Stanford University (Bob West and Jure Leskovec) aimed to understand and improve link coverage in Wikipedia across languages. In collaboration with this team, we submitted a workshop to ICWSM '15 that was accepted in the conference program and will be held in Oxford on May 26, 2015.
We kicked off a new IEG-funded project to bring a machine learning based revision scoring service to Wikimedia Labs with User:とある白い猫 and User:He7d3r.
We published an overview of the most edited articles in 2014 (press). We published an open data corpus from the Article Feedback pilot in collaboration with the Product team. The corpus contains over 1.5 million messages posted on three major Wikipedias (English, French, German) over the course of a year.
We hosted our monthly showcase with a presentation on mobile readership by Oliver Keyes and a guest talk on flu forecasting with Wikipedia data by Reid Priedhorsky (Los Alamos National Laboratory).
The Kiwix project is funded and executed by Wikimedia CH.
- The Google Code-in project has started and we are part of it, we propose a good dozen of tasks related to Kiwix for Android.
- After two years since the 0.9rc2, we have finally released Kiwix for PCs (Windows, OSX, Linux) version 0.9. This new release brings a lot of small improvements and bug fixes which should really improve the global user experience.
- Most of our resources were used this month to improve mwoffliner, our solution to build a ZIM file (offline version) of any Mediawiki. These efforts to improve the performance, the features and the usability start to pay, We have built more than 400 ZIM files this month (trend increasing) and third party projects start to use it to provide their own ZIM files (like here or here).
- On the operational side, we have started to setup a ZIM building farm in the Wikimedia labs. Work is ongoing but the goal is to be able to generate soon, one time a month at least, ZIM files for all our projects.
The Wikidata project is funded and executed by Wikimedia Deutschland.
- In December the Wikidata team released a number of new features. These are:
- language fallbacks (so you see information in a different language if it is not available in your primary language)
- statements on properties (this is important for mapping our properties to other databases for example)
- a new datatype for linking to properties (this is necessary to say that one property is the inverse of another property for example - like "father" and "child")
- performance improvements.
- In addition we provided support for the work on query functionality for Wikidata. Your input is requested for a new feature: article placeholders powered by Wikidata.
- It was announced that Freebase will be shut down in favor of Wikidata. Lydia wrote a blog post about scaling Wikidata over the next year.
Future[edit]
- The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.