In September we have setup (backup) replication of most project data, including core databases and external storage. Work on Swift images and system backups was still ongoing into October. Essential system infrastructure such as an installation server, DNS, LVS, NTP etc. has been deployed as well.
Tampa data center
We started the last push to get the remaining services & systems out of our Tampa data center, with a deadline for shutdown of all systems on October 1st. The remaining services included PDF generation, mail servers, noc.wikimedia.org and LDAP.
Labs metrics in September:
Number of projects: 146
Number of instances: 415
Amount of RAM in use (in MBs): 1,996,288
Amount of allocated storage (in GBs): 20,435
Number of virtual CPUs in use: 977
Number of users: 4,083
Wikitech (the Labs web interface) is now managed via the standard WMF deployment system. This should allow for more frequent MediaWiki updates and overall greater stability.
The last historic remaining dependencies on our old Tampa datacenter (e.g. LDAP and Labs DNS backup servers) were finally stamped out and replaced with dependencies on Dallas hardware.
One of the labs virtualization hosts (virt1006) was suffering intermittent problems, so all affected instances were migrated to other hosts in order to stave off possible future disaster. Consequently, Labs is a bit short on virtualization space, but new hardware procurement is under way.
Several long-unused instances and projects were cleaned up in order to free up more space.
The last of the ToolLabs replica DB servers was upgraded to MariaDB 10.
In September, the team working on VisualEditor expanded browser support, improved some features, and fixed nearly 60 bugs and tickets.
Users of Internet Explorer 10, who we were previously preventing from using VisualEditor due to some major bugs, will now be able to use VisualEditor; this follows on from Internet Explorer 11 support last month. When editing a template with a required field, VisualEditor now warns you to avoid leaving it blank, and you can now create auto-numbered links using VisualEditor.
Improvements and updates were made to a number of interface messages as part of our work with translators to improve the software for all users, based on feedback from users and user testing. We made progress on table structure editing and auto-filled citations, both of which will be coming soon.
In September, the Editing Team made substantial progress on front-end standardisation, as well as the work on VisualEditor which is reported separately. The team welcomed Bartosz "MatmaRex" Dziewoński as a new team member, and existing student member Moriel Schottlender converted to full-time status.
The team's work on front-end standardisation is focussed on improving libraries and infrastructure, and in particular, the OOjs UI library. This included the creation of a MediaWiki theme in collaboration with the Design team, which can be explored in the online demo; this will be deployed into MediaWiki's use of OOUI in the next few weeks. A number of bugs were fixed, including working around window and popup sizing, over-flow item placement, and working around some browser bugs in Firefox and Safari. The code documentation has a number of minor issues corrected, and the build process was extended to create a minified distribution. The OOjs library was updated to fix a minor bug in oo.Compare, with a new version (v1.1.1) released and pushed downstream into MediaWiki, VisualEditor and OOjs UI.
The TemplateData extension now supports the "autovalue" parameter property, a wikitext value that a parameter can be set to have inserted by default if desired. Also, the specification for TemplateData was re-written to be clearer and more consistent. Next month the TemplateData GUI editor will be made available on all Wikimedia wikis.
In September, we continued to fix bugs, upgraded libraries, and made additional progress towards improving compatibility with PHP parser + Tidy rendering. Specifically, Parsoid's paragraph wrapping now targets the PHP parser + Tidy output rather than PHP parser output. We also continued to update Parsoid's CSS / rendering to more closely match the current rendering. We also improved Parsoid's robustness handling edge case scenarios (pathological backtracking, parsing of very large wikitext tables). Part of the Parsoid team was also busy with launching the PDF rendering service which was successfully launched end of September.
September saw a lot of activity on the RESTBase storage and API service. A new 'pagecontent' composite bucket type using revisioned blob buckets was introduced. This uses the by-now fairly rich table storage backend to provide functionality similar to MediaWiki's revision table, and supports any number of revisioned types of content (like HTML, wikitext, JSON metadata) associated with each revision.
In September, the Flow team enabled new test pages on French WP and Hebrew WP. The French test is for the Forum des Nouveaux, a Help space for new contributors (similar to the Teahouse on English WP). The Forum des Nouveaux hosts reached out to the Flow team after Wikimania, excited to try out the new discussions system. The Hebrew WP test is helping the team diagnose problems for Right-to-Left languages, and general i18n issues.
The team also refined the new Echo notifications functionality, with lots of feedback from contributors on Mediawiki.org and En.wp. New topic notifications are now bundled in Echo, and we fixed several bugs related to the behavior of the Alerts and Messages tabs, and getting excess mention notifications.
In September, the Mobile Apps Team released a new version of the iOS app containing the Nearby feature which shows you articles about things that are near your location, and a references panel that pops up whenever you tap a reference. The team also released an iOS 8 compatibility build to market. The team also spent time performing code quality improvements and refactoring on both the iOS and Android apps.
This month the mobile web team focused on the first prototype of WikiGrok, a new contribution feature that asks users who are reading Wikipedia articles to help add Wikidata that is missing about the article subject. Over the course of the month, we built and user-tested the first experimental interface for allowing users to input Wikidata: a simple binary question mode that provides the user with a suggested occupation on biographies that are missing this information in Wikidata but contain a possible occupation in the Wikipedia article. In this early test phase we are storing the replies in a separate database, not pushing to Wikidata. We plan to add suggestions for more Wikidata fields and test this version against a slightly more complex tagging interface in beta in October.
During September 2014, the team built more Partners Portal architecture, including Graph extension integration components for eventual display of aggregate statistics to zero-rating partners (it already works and is being reviewed in house). The team also grew support for dynamic zero-rating banners while enhancing JSON configuration extension code and issuing bugfixes. Additionally, the team shrunk the size of the Wikipedia favicon to reduce bandwidth usage by users across the web. And on the partner side, we launched Telenor Myanmar in September.
Finally, the team welcomed its newest software engineer, Jeff Hobson, to the Wikimedia Foundation!
The CLDR extension was updated to version 26 and entries identical to CLDR were removed from LocalNamesEn.php. The team made RTL fixes in core, Echo and Wikibase, and tested Flow for RTL support. Maintenance of the Translate extension continued, and the performance of translation memory was improved on ElasticSearch with the help of Nik Everett
In September we worked to mitigate the performance bottleneck that we found in August. We found there to be no silver bullet but used the information we learned to pick and order appropriate hardware to handle the remaining wikis. We also implemented out significantly improved wikitext Regular Expression search.
In October we've begun rolling out the wikitext Regular Expression search and received some of the hardware we need to finish cutting over the remaining wikis. We believe we'll get it all installed in October and cut the remaining wikis over in November.
In September, the team wrapped up the feature development for SUL finalisation. One part of the work (the steward end of the rename request form) is outstanding and will be finished in October. In October, the team is planning to proceed into deployment and testing of the features.
We also continued our code refactoring for the UploadWizard, and started to collect metrics for an upload funnel analysis, to find out how many users drop out at each step of the upload and where failure is occurring, so we can prioritize bug fixes. For more information about our work, join the multimedia mailing list.
The Vital Signs dashboard is now live. We are calling it “Vital Signs” because it will eventually display content and readership metrics, not just Editor Engagement metrics.
Vital Signs was presented at the Analytics Quarterly Review as well as the October WMF Metrics meeting.
This month we onboarded Ellery Wulczyn as the newest addition to the Research & Data team. Ellery recently finished a Computer Science Masters program at Stanford and joins us as a full-time research analyst after completing a summer fellowship with University of Chicago's Data Science for Social Good program. His focus at WMF is going to be fundraising research and analytics. Welcome, Ellery!
We made progress in our work with our partner Bookeen to get an e-ink reader able to read Wikipedia offline. We managed to get a first version of the device firmware working, and it will be tested in the field as part of the Malebooks pilot deployment.
As a consequence of a bug fixing sprint with Parsoid and Wikisource developers at Wikimania, we were finally able to generate usable ZIM files from Wikisource (example with fr.wikisource.org).
In September, the Wikidata team focused on improving performance, doing groundwork for the new user interface design, and making it possible to track where data from Wikidata is used. Next to that, they worked on tests and prepared for a week-long meeting with the WMF multimedia team and volunteers to discuss and plan structured data support for Wikimedia Commons.
The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.