Work to refactor and modularize our Puppet repository continues: this month, lots of dead code was removed, and some tiny miscellaneous classes aggregated with more relevant components. Work on git-deploy has also restarted this month. Changes were made to make git-deploy easier to configure and to make initial setup of new repositories and setup of new minion targets completely automated.
Many of the services within the Tampa data center have already migrated to EQIAD, however there remain several smaller, unique, or in some cases orphaned services that we still need to document or scope prior to the closure of this center. Several of these services may no longer be required, and we expect there to be some discussion about how they are migrated or maintained going forward. Additionally, several of these systems need to be moved to the new secondary data center (see below), and will be waiting until infrastructure is in place to do so. Our goal is to try to have these systems moved before the end of 2013, but we'll continue to have equipment in this location for as long as necessary to ensure stability of our network.
For EQIAD, the ordering process is underway to complete our fourth row of machines, and ensure we have capacity to take in systems that will be arriving from the sunsetting of the Tampa data center, as well as handle our expected growth.
For ULSFO, after a long initial setup, initial bootstrapping and configuration of the systems is finally underway. Over the next several weeks, we will be configuring, testing, and redirecting traffic at this location.
Lastly, work has begun on a definition and RFP process for a new, secondary data center, likely on the west coast of the US. We will send further updates on this project once our RFPs are complete and we begin the selection process. Our hope is to have this facility ready to take systems from Tampa by the end of 2013.
The DNS infrastructure of the Labs has been overhauled and much improved. The hardware switch to replace Labs' NFS server unreliable hardware is ready, and should be enabled this week. Yuvaraj Pandian has created and deployed a new instance proxy with an OpenStack-style API. The new proxy is in use for a small number of instances right now, but will be expanded to most instances in the future. The new proxy uses nginx with Lua code to read its configuration of virtual hosts from redis and can handle arbitrary URLs to arbitrary back-ends.
In September, the VisualEditor team continued their work to improve the editor and roll it out to additional wikis. The deployed version of the code was updated four times (1.22-wmf16, 1.22-wmf17,1.22-wmf18 and 1.22-wmf19). The focus in the team's work this month was to continue to improve the stability and performance of the system, fix a number of bugs uncovered by the community, and make some usability improvements.
We fixed a few bugs reported in production, added performance stats to our RT-testing framework (and discovered a couple bugs and fixed them as a result) and did some long-standing cleanup work in our codebase. September also saw the all-staff meeting at the WMF offices in San Francisco which gave us the opportunity to work in person and discuss some proposals. We planned out an implementation strategy for language variant support, and started researching and experimenting with HTML storage options which is required for a number of projects in our roadmap.
In September, we released Notifications on more Wikipedias, such as the Dutch, Hebrew, Japanese, Korean, Spanish, Ukrainian and Vietnamese. Fabrice Florin and Keegan Peterzell managed community relations for these new releases, and are reaching out to more projects. Our next deployments will take place every other Tuesday. Developer Benny Situ was responsible for these deployments and fixed a number of bugs, with the help of Erik Benhardson and Matthias Mullie. Community response has been very positive so far, across languages and regions. For each release, we reached out to community members weeks in advance, inviting them to translate and discuss the tool with their peers. As a result, we have now formed productive relationships with volunteer groups in each project, and are very grateful for their generous support. To learn more, visit our project hub, read the help page and join the discussion on the talk page.
This month, we continued back-end work on the Flow first release – integrating with the recent changes table (to ensure that users will be able to monitor Flow boards via the watchlist and Special:Recentchanges, in the same way they monitor wiki pages), mentions and notifications, and an early experiment with VisualEditor-enabled posting. We also kicked off a sprint to create a new visual design treatment for the board and discussions that will work across desktop and mobile platforms. We are aiming to implement this design next month, in preparation for several rounds of new user and experienced user feedback before the first onwiki release.
In September, the Growth team (formerly known as Editor Engagement Experiments, or E3), primarily worked on the onboarding new Wikipedians project. In particular, this included the creation and deployment of two new guided tours to teach any new user how to make their first edit, using wikitext or VisualEditor. The guided tours extension was also deployed to the following language editions of Wikipedia: Catalan, Hebrew, Hungarian, Malay, Spanish, Swedish, and Ukrainian.
This month, the team mostly focused on preparing for the upcoming English fundraiser. Planning began for periodic tests throughout October, which will help determine the launch date and other aspects of our fundraising efforts in November and December.
In September, we continued to expand our multimedia team and updated our multimedia plan for the coming year (see slides). Mark Holmquist continued development on the Media Viewer to improve the image viewing experience, based on designs by May Tee-Galloway and Jared Zimmerman. We also made good progress on the Beta Features project, which will invite users to test, give feedback, and use a range of new features in real-world settings. We aim to have first beta versions of both products ready by the end of the October. New employee Bryan Davis started work on several multimedia platform bugs, and Summer contractor Jan Gerber completed his work on the TimedMediaHandler extension. To discuss these features and keep up with our work, we invite you to join the new multimedia mailing list. Last but not least, we are also recruiting for a senior software engineer position on our team.
Although this activity is still officially on hold, several bug fixes were committed this month by community members. The GSoC project to implement a simple Bayesian Filter extension was successfully completed.
In September, we expanded the new CirrusSearch back-end to a number of wikis. Italian Wiktionary, Catalan Wikipedia and English Wikisource are all running CirrusSearch now. Additionally, we deployed to all "closed" wikis. Further feature refinement and bugfixing are ongoing, with roughly 2 to 3 deployments a week.
The team improved the user interface of OAuth and deployed these changes to mediawiki.org and test.wikipedia.org. We hope to test and refine the extension with third party developers, and subsequently deploy to all wikis. An initial review of Extension:OpenID was performed, and several issues were brought to the attention of the extension maintainer. Several bugs with CentralAuth/SUL were also fixed.
This month, we wrapped up Rachel Thomas' Outreach Program for Women internship successfully. Rachel helped us extend our browser test coverage of VisualEditor. Besides our ongoing collaboration with Wikimedia Foundation development projects, we are also engaging the greater community on the QA mailing list, where we discuss both code contributions and general QA topics.
This month saw significant improvements to both coverage and speed in our tests for VisualEditor. We are collaborating with the Language team on browser tests for the UniversalLanguageSelector extension and Translatewiki.net. We created our first tests for the new Flow feature and are in the process of supporting Flow fully in a reference test environment. We presented yet another of our ongoing series of training sessions, this one live in San Francisco.
Guillaume Paumier wrapped up work on supporting the deployment of VisualEditor, and resumed regular activities like preparing the Tech newsletter and ongoing communications support for the engineering staff.
Together with XWiki and Tiki, we submitted a Wiki devroom proposal for FOSDEM, the biggest open source source conference in Europe. We are also preparing a proposal for a stand, lead by volunteers at the nascent Wikimedia Belgium chapter. The overall goal is to achieve a good MediaWiki & Wikimedia tech gathering in Brussels next February.
We are also supporting the organization of the MediaWiki Architecture Summit in San Francisco on 23-24 January, 2014.
The team has been focused on smaller but more important work items this month, including enhancement to Wikimetrics, Grantmaking and Program Developments graphing infrastructure and fixing some long-standing Limn bugs. On the infrastructure side, our collaboration with Ops has the Kafka middleware project moving along nicely. The all-staff meeting and travel schedules definitely impacted our throughput this month.
Two notable accomplishments should be called out: our Hadoop environment is now 100% free software, as we swapped out a proprietary JDK for OpenJDK 7. We also spent a lot of time on our engagement processes and planning for our first combined quarterly review in October, and made significant process on our hiring goals.
This month, Aaron Halfaker joined the research team as a full-time employee. We started to reorganize the team structure and engagement model in coordination with the Analytics developers. We performed a survival analysis of new editors in preparation for new experiments led by the Growth team, and worked with the team to iron out the data collection and experimental design for the fortcoming iteration of GettingStarted.
We drafted a proposal to host tabular datasets in a dedicated namespace and solicited feedback from interested parties (particularly the Wikidata community). We also started fleshing out the Labs2proposal, an outreach program for academic researchers and community members, launched at Wikimania 2013 in Hong Kong. We co-hosted the second IRC research office hours and prepared for the first Wikimedia research hackathon, an offline/online event to be held in various locations worldwide on November 9, 2013.
In September, the Wikidata team mainly concentrated on the sister projects Wikimedia Commons and Wiktionary. For Wikimedia Commons, we added the ability to store interwiki links in one central location (Wikidata) together with the ones for Wikipedia and Wikivoyage. For Wiktionary, we published an analysis of all existing proposals for the integration of Wikidata and Wiktionary.
The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.