Roadmap

Background: In recent years, Wikimedia Foundation's engineering planning has primarily taken place in a combination of internal goals documents and public project-level documentation. The purpose of this page is to establish a practice of sharing holistic engineering goals, initially primarily with a short term horizon, as a public document with open participation. See also:
 * Product Whitepaper for Wikimedia Foundation product priority justifications; Strategic Plan for long-term WMF goals, some of which are engineering goals

Item syntax:
 * Project name
 * [Qualifier keyword] Action item [Owner(s)] [Completion keyword]

Examples:
 * New unicorn engagement
 * [Needs consensus] Deploy unicorns [Brion/Tim]
 * Deploy unicorns [Brion/Tim] [DONE]

Continuous Responsibilities

 * Code review
 * RT review / ops tasks
 * Community extension review and deployment
 * Bug triages and general bug fixes
 * New extension deployment (Review queue)
 * Labs maintenance
 * Test infrastructure maintenance
 * Recruitment and encouragement of volunteers
 * Monthly report and other engineering communications

Technical Operations

 * Swift @ Tampa upgrade from 1.5 to 1.74 [Faidon/Ariel] [frontend servers - done; backend be part of the swap out process]
 * Ceph evaluation [Asher / Mark / Faidon] [on-going]
 * Copy thumbnails from Swift  to nas01 [Aaron/Faidon/Mark] [Team decided not doing it]
 * Productionize 2 extra parser cache servers (pc2, pc3) @ Tampa [Asher]
 * Deploy Varnish for upload@ESAM [Mark]
 * Swift Servers swap out (C2100 to R720XD) @ Tampa [Chris, Ariel [ in-progress ; expected to complete in end January]
 * Provision Memcached servers @ Eqiad [Peter] - done
 * Provision parser cache servers @ eqiad [RobH, Peter, Asher] - done
 * Git deployment system - alpha release [Ryan] - done
 * provision & deploy Parsoid servers for Alpha release [RobH] - done
 * Test Git-deploy


 * Labs
 * Push OpenStackManager changes to show SSH fingerprints for instances [Ryan Lane] (not a high-priority item, will probably be December 2012 or  later)
 * Add DNS support to Nova (in essex, but we must migrate to using it)
 * Add Gluster support to Nova (Gluster plugin is mostly written but we don't really have plugin support until folsom, so will happen in Q3 when  we are ready to upgrade to Folsom)
 * Migrate nfs home directory to glusterfs [done]
 * Discussion with Sumana re product management, Dec 21

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * Iteration on Visual Editor for Dec release [DONE]
 * limited production release on English Wikipedia [DONE]
 * IE in VE [IN PROGRESS]
 * bug fixing on Visual Editor [ONGOING]
 * [Parsoid testing framework for semantic/superficial regressions|http://parsoid.wmflabs.org:8001] [IN PROCESS]
 * Node.js parsoid to completion [ONGOING]
 * limited production release on French, Hebrew Wikipedia [IN PROGRESS]


 * New Pages Feed (fka Page Triage) (Kaldari, Benny, Nischay)
 * Descriptive statistics on NPF (revised dashboard on toolserver) [Dario] [DONE]
 * Bug fixes (no more feature development) [ONGOING]


 * Echo_(Notifications) (Kaldari, Benny, Werdna, Luke Welling)
 * Continued development of prototype for Mediawiki.org [ONGOING]
 * Deployment on mediawiki.org [Dec 18] [DONE]
 * UI and bugfixing of Echo [ONGOING]
 * HTML e-mail [IN PROCESS]
 * Global user changes [JamesF] [IN PROCESS]
 * i18n issues on Echo
 * (see Performance engineering for infrastructure prep work:)
 * queueing & Echo
 * Database sharding & Flow


 * Article feedback (Matthias)
 * sharding (see Database sharding) [Matthias] [CANCELLED], alternative implementation strategy in development
 * dewiki pilot launch [DONE]
 * performance/db related changes to AFTv5 queries [Matthias] [IN PROCESS]
 * i18n messaging update [IN PROCESS]
 * Other minor changes prior to full English Wikipedia deployment [Jan]


 * E3 (Ori, S, MattF, Luke)
 * Clicktracking/Analytics
 * Funnel tracking of Community Portal [ONGOING]
 * Funnel tracking of Onboarding [IN PROCESS]
 * Account Creation User Experience
 * Data model [IN PROCESS]
 * Productization of ACUX [IN PROCESS]
 * Data analysis once experiment is complete [IN PROCESS]
 * Onboarding
 * Extensionize guided tour [DONE]
 * Javascript bugs in guided tour [IN PROCESS]
 * Deploy first version of onboarding to production [DONE]
 * Continue working with SuggestBot author to develop list of tasks -- [Bot Approved]
 * get Guided Tour ready for deployment [IN PROCESS]
 * Other Account Creation Funnels
 * Sue's donation Thank you email [DONE]
 * 22k Donor Survey email [IN PROCESS]
 * Contribution banner [IN PROCESS]


 * MicroDesign
 * other edit window templates [IN PROCESS]
 * (see Agora)


 * List of other projects on radar
 * ResourceLoader2
 * LiquidThreads 3.0

Fundraising (Katie,Adam,MattW,PeterG)

 * Fundraiser - 5 countries [IN PROGRESS]
 * Thank You e-mail and flow (with E3 team) [DONE]

Mobile

 * Mobile Web
 * Beta
 * photo upload [DONE]
 * Watchlists add and view (recent changes view) [DONE]
 * Better table support [DONE]
 * lots of bug fixes [DONE]
 * Editing [DONE]
 * Account Creation [IN PROCESS]
 * Reader-friendly Watchlist [IN PROCESS]
 * Last Modified [IN PROCESS]
 * History [IN PROCESS]
 * Alpha [IN PROCESS]
 * User Testing of Beta Editing, next steps TBD
 * Production
 * Push beta features to product (TBD)
 * BugFixes [DONE]
 * Variant Support [DONE]
 * Account creation [DONE]
 * GeoData EN Wiki [DONE]

MediaWiki Core

 * Datacenter migration (see TechOps section above) [Tim, Aaron]
 * Git/Conversion (Chad)
 * LDAP fixes for our Gerrit use (current patch) [Chad] [partial]
 * More work on behalf of upstream (up to patchset 10 now), but needs approval from primary maintainer
 * MediaWiki 1.20/Roadmap
 * Bi-weekly deployment cycle continues
 * MediaWiki 1.21/wmf5 -- [DONE]
 * MediaWiki 1.21/wmf6 -- [DONE]
 * Holiday break
 * Continuous integration
 * Running unit tests for whitelisted users only [Antoine] [DONE]
 * Wikidata deployment
 * Deploy Wikibase client to test2 [Chad] -- [DONE]
 * Lua scripting [Tim, Chad, Brad]
 * Additional APIs to replicate parser functions [Tim, Chad, Brad] [partial]
 * Security auditing and response
 * Training documentation sprint [CSteipp] [DONE]

Multimedia

 * TimedMediaHandler
 * Deploy new async upload API support to UploadWizard [Aaron, Jan] -- [DONE]
 * Bugfixing deployment related issues [Jan] -- ongoing

QA

 * QA and testing
 * Data center migration
 * Test plan and automated testing training -- [Chris M, Peter]
 * Visual Editor
 * Visual Editor to be deployed to test2: https://bugzilla.wikmedia.org/show_bug.cgi?id=43018
 * High priority: some automated regression tests, quick smoke tests across browsers especially for Internet Explorers - ongoing
 * E2
 * Creating browser regression tests for Article Feedback [DONE]
 * Browser testing
 * Mobile
 * Create first Mobile browser tests [DONE]
 * Mobile team working on bringing Mobile Frontend to beta labs: https://bugzilla.wikimedia.org/show_bug.cgi?id=40605 [Max, Arthur]
 * Beta labs (beta cluster)
 * Continue refining beta labs performance [Antoine/Chris/Željko]
 * Continuous integration
 * Ported reporting frameworks from raw RSpec to Cucumber (see below) [DONE]
 * Make Cloudbees Jenkins public/read-only [DONE]
 * Publish information from Cloudbees Jenkins to WMF Jenkins [NOT DONE Zeljko and Chris]
 * Fixing glitches with file handling in Jenkins (works locally) [DONE]
 * Form one ATDD user group (Acceptance Test Driven Development) to work with test backlog test backlog page.  [ONGOING, Chris and Quim]
 * Coverage goals: regression tests from backlog page, extensions managed on beta. [ONGOING]
 * Socializing & training community in writing automated tests -- ongoing [Chris & Quim]

Analytics
See also: Analytics/Roadmap

Kraken

 * Product Codes: Email stakeholders and gather feedback [dsc] (Dec)
 * Ensure Mobile is sending data with a product code!! [dsc] (Dec)
 * EventLogging extension integration
 * Make sure all event data goes into Kraken (I think it may only be esams at the moment, not sure) [otto] (Dec)
 * Divvy up TODOs [otto + dsc + ori] (Dec)
 * Set up Maven / Somatype Artifact Repository (continuous integration) [dsc] (Dec)
 * WMF Maven parent pom [dsc] (Oct) [DONE]
 * Walkthrough to get up and running with Maven in Eclipse [dsc] (Dec)
 * Core Jobs: aggregation, bucketing [dsc, DvL, others] (Dec)
 * Import Mediawiki tables using Sqoop [dvl] (Dec)
 * Create tool to generate sqoop import statements and oozie workflow documents (Sqoopy) [dvl] (Dec) [DONE]
 * Finetune Kraken configuration (Hue, Hive, Oozie) [dvl + ottomata + dsc] (Dec)
 * Puppetize Kraken [otto] (Ongoing)

Limn

 * Regular deploys to http://dev-reportcard.wmflabs.org [dan] (Dec)
 * Mirror GitHub to Gerrit [dsc] (Dec)
 * Support the Global Dev dashboard [evan] (Ongoing)
 * Support the Gerrit Stats dashboard [diederik] (Ongoing)

Legacy Log Collection

 * Add support for new domain names in webstatscollector (blog, etc) [diederik] (Dec) [DONE]
 * udp2log filters
 * Update filters for Wikipedia Zero [otto] (Ongoing)

WikiStats

 * Setup Jenkins support [stefan] (Dec) [DONE]
 * Wikistats for editors [stefan] (Dec) [DONE]
 * Decouple wikistats from stat1 (so you can run it locally) [stefan] (Dec) [DONE]
 * Fix country related mobile pageview bugs [stefan / diederik] (Dec) [DONE]
 * Git migration [ezachte/diederik] (Nov/Dec) [DONE]
 * Reduce backlog regarding Wikistats traffic (squid etc) scripts [diederik / stefan] (Oct)
 * Repair data errors in wikistats, and add process for checking data integrity [stefan / diederik /ezachte] (Sept)
 * Make wikistats more robust (MoM validations) [stefan] (Oct)
 * Add Blackbox testing to WikiStats [diederik + stefan] (Oct)

Infrastructure

 * Access/support requests for stat1, stat1001 [otto] (Ongoing)
 * Maintenance of oxygen/emery/locke [otto] (Ongoing)

Data Releases

 * Start pushing datasets to AWS [diederik] (Nov) [DONE]
 * Finalize scripts to massively compact dammit.lt data [erik] (Oct)
 * Deploy stats.grok.se on stat1001 [diederik/dario/andrew] (Dec/Jan)
 * Create Data Release Practices Task Force [diederik/dario] (waiting for feedback from Dario) (Sept)
 * Publish Monthly Report Card -- deal with monthly data processing irregularities, perform correction/validation [ezachte + diederik + dsc] (Ongoing)

Engineering Community Team

 * Bug Wrangling [Andre + volunteers]
 * Improving Bugzilla workflow -- ongoing [Andre]
 * Updating BZ maintenance documentation -- ongoing [Andre]
 * Prioritizing unprioritized bugs [Andre]
 * Mentorship programs - ongoing [Quim, Sumana, Andre]
 * Summer of Code 2012: Pushing on merges -- ongoing [Sumana]
 * Outreach Program for Women - ongoing [Quim, Sumana, Andre]
 * LevelUp [Sumana] -- ongoing - socialized internally, soon to launch publicly: target Dec 20
 * Volunteer coordination and outreach
 * Event planning & followup -- ongoing [Sumana, Quim]
 * Bangalore followup [Mobile & language engineering teams]
 * Grace Hopper followup [Sumana]
 * OpenStreetMap and 2013 Developers Meeting meetings -- consultation/planning [Sumana/Quim]
 * Planning online or in-person QA trainings for the community -- ongoing [Quim]
 * Planning community testing events -- ongoing [Quim]
 * Encouraging technical user groups -- ongoing [Quim]
 * Implicit knowledge transfer from Sumana to Quim -- ongoing with target of end of December
 * Technical communications [Guillaume]
 * Tech Ambassadors Network [Guillaume] -- ongoing on meta
 * Review of consultation results & creation of new experiment - decision coming Dec 20
 * Volunteer product management program -- ongoing [Sumana]: target for new intake, end of December

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * IE in VE [IN PROGRESS]
 * bug fixing on Visual Editor [ONGOING]
 * [Parsoid testing framework for semantic/superficial regressions|http://parsoid.wmflabs.org:8001] [IN PROCESS]
 * Node.js parsoid to completion [ONGOING]
 * limited production release on French, Hebrew Wikipedia [IN PROGRESS]
 * 5-month plan for VE [late Jan]
 * assessment for next steps in Parsoid [mid Jan]
 * Echo_(Notifications) (Kaldari, Benny, Werdna, Luke Welling)
 * Continued development of prototype on Mediawiki.org [ONGOING]
 * UI and bugfixing of Echo [ONGOING]
 * HTML e-mail [Luke] [IN PROCESS]
 * Global user changes [JamesF] [IN PROCESS]
 * i18n issues on Echo
 * JobQueue abstraction (zmq) [Luke] [IN PROCESS]
 * JobQueue abstraction (Redis) [Luke]


 * [Flow] (Werdna, JamesF, Brandon, Matthias)
 * Kickoff
 * reach out to Wikia [Terry, JamesF]


 * Article feedback (Matthias)
 * performance/db related changes to AFTv5 queries [Matthias] [IN PROCESS]
 * code review for performance/db patch [pending]
 * i18n messaging update [IN PROCESS]
 * Other minor changes prior to full English Wikipedia deployment [Jan]
 * refactored deployment on enwiki + dewiki pilot (may involve db pruning) [Late Jan/earlyFeb]
 * vertically partitioned database deploy [Feb]


 * New Pages Feed (fka Page Triage) (Kaldari, Benny, Nischay)
 * Bug fixes (no more feature development) [ONGOING]


 * E3 (Ori, S, MattF, Luke)
 * Clicktracking/Analytics
 * Funnel tracking of Community Portal [ONGOING]
 * Funnel tracking of Onboarding [IN PROCESS]
 * Account Creation User Experience
 * Data model [IN PROCESS]
 * Productization of ACUX [IN PROCESS]
 * Data analysis once experiment is complete [IN PROCESS]
 * Onboarding
 * Javascript bugs in guided tour [IN PROCESS]
 * Continue working with SuggestBot author to develop list of tasks -- [Bot Approved]
 * get Guided Tour ready for deployment [IN PROCESS]
 * Other Account Creation Funnels
 * 22k Donor Survey email [IN PROCESS]
 * Contribution banner [IN PROCESS]


 * MicroDesign
 * other edit window templates [IN PROCESS]
 * (see Agora)

Technical Operations

 * Datacenter migration (eqiad)
 * Updating the scap scripts to deal with multi-datacenter
 * change to git-deploy?
 * deploy memcached@eqiad
 * deploy parser cache@eqiad
 * deploy apache app servers@eqiad
 * deploy API servers@eqiad
 * deploy jobrunner servers@eqiad
 * database failover from Tampa to Eqiad
 * deploy parsoid servers@eqiad


 * Deploy Etherpad Lite [Daniel/RobH]

Mobile

 * Quarterly planning session with team: focus on photo uploads (e.g., commons app, productizing mobile photo uploads)
 * Getting MobileFrontend onto beta labs
 * Mobile Web
 * Beta
 * Mobile diff
 * Watchlist CTA
 * User Testing of Beta Editing, next steps TBD
 * Production
 * Push beta features to product (TBD, but currently shooting for):
 * Watchlist (reader + recent changes views)
 * Watchlist CTA
 * Account Creation
 * Login
 * History
 * Better resource loader support

MediaWiki Core

 * Datacenter migration (see TechOps section above) [Tim, Aaron, Sam, ChrisS]  39106
 * git-deploy migration (43338) and file path issues for eqiad migration (43340) [Brad, Antoine, Sam, ChrisS]
 * Remove production MediaWiki dependency on NFS (43495) [Aaron, Tim]
 * Git/Conversion (Chad)
 * LDAP fixes for our Gerrit use (current patch) [Chad]
 * Upgrade to 2.6 [Chad]
 * Replace gitweb with GitBlit [Chad]
 * MediaWiki 1.20/Roadmap
 * MediaWiki 1.21/wmf7
 * MediaWiki 1.21/wmf8
 * Admin tools development [Chris]
 * Build and deploy Global RenameUser extension [CSteipp]
 * Lua scripting [Tim, Chad, Brad]
 * Install Lua (Scribunto) on other Wikimedia wikis ( see Goals)
 * Detailed deployment plan [TBD]

Multimedia

 * TimedMediaHandler
 * Bugfixing (Jan)

QA

 * QA and testing
 * Browser testing
 * Beta labs (beta cluster)
 * Add specific reviewer in Gerrit on SQL changes [IN PROCESS] [Antoine]
 * Continuous integration

Analytics
See also: Analytics/Roadmap

Kraken

 * Storm pom [dsc] (Jan)
 * Storm ETL
 * Get Storm set up [dsc + otto] (Dec) [DONE]
 * Storm ETL bolt for GeoIP [dsc + otto] (Jan)
 * Storm ETL bolt for HDFS Import [dsc + otto] (Jan)
 * HDFS Import enforces product-code directories and permissions [dsc + otto] (Jan)
 * Storm ETL bolt for Anonymization [dsc + otto] (Jan)
 * Storm ETL bolt for Kafka Checkpointing [dsc + otto] (Jan)
 * Puppetize Kraken [otto] (Ongoing)

Limn

 * Refactor charting to use d3 [dan + dsc] (Jan)
 * Feature Parity with Dygraphs (plus bugfixes, etc) (Jan)
 * Bugfixes (like Save-As, UI Error Notifications, ...) [dan] (Jan)
 * Migrate Dario's dashboards to Limn [dsc + dan] (Jan)
 * Improve Limn wiki, docs, & guides (esp those shameful screenshots) [dan] (Jan)
 * Coke ( for Coco) task to create symlinks into   from an existing data repository (such as, say, analytics/reportcard/data) [dsc] (Jan)
 * Coke task to download and setup dummy testing data for ease of development [dsc] (Jan)
 * UI support for remote datasets via proxy [dsc + dan] (Jan)
 * Deploy reportcard / gerrit-stats on stat1001 (aka, "the Debian packaging discussion") [otto + dsc] (Jan)
 * Support the Global Dev dashboard [evan] (Ongoing)
 * Support the Gerrit Stats dashboard [diederik] (Ongoing)

Legacy Log Collection

 * udp2log filters
 * Update filters for Wikipedia Zero [otto] (Ongoing)
 * Filter by X-Carrier headers. [otto + asher + diederik] (Jan) [Waiting for Fundraiser to finish]'
 * udp-filter to filter by http status. [otto] (Jan) [Waiting for Fundraiser to finish]

Infrastructure

 * Access/support requests for stat1, stat1001 [otto] (Ongoing)
 * Maintenance of oxygen/emery/locke [otto] (Ongoing)
 * Migrate Reportcard off Labs onto stat1001 -- reportcard.wikimedia.org [otto + dsc] (Jan)

Data Releases

 * Blogpost about what awesome stuff you can do with the AWS datasets [diederik] (Jan)
 * Deploy stats.grok.se on stat1001 [diederik/dario/andrew] (Dec/Jan)
 * Create Data Release Practices Task Force [diederik/dario] (waiting for feedback from Dario) (Sept)
 * Publish Monthly Report Card -- deal with monthly data processing irregularities, perform correction/validation [ezachte + diederik + dsc] (Ongoing)

Engineering Community Team

 * Bug Wrangling [Andre + volunteers]
 * working especially on reducing backlog of unprioritized issues
 * mentoring Valerie in bug triage
 * Mentorship programs - ongoing [Quim, Sumana, Andre]
 * Outreach Program for Women kickoff
 * LevelUp kickoff
 * Volunteer coordination and outreach
 * Technical communications [Guillaume]
 * Kickoff of first tech collaboration experiment
 * Misc Engineering community proposals' followup [Sumana]
 * Code of Conduct [Sumana] -- in process - target 11 Jan

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * experimentation into C/C++ version


 * Echo_(Notifications)
 * Global preferences for echo delivery [Deferred to 2013]
 * centralization and cross wiki notification delivery [Deferred to 2013]


 * Article feedback (Matthias)
 * Full deployment to English Wikipedia [2013]

Parking lot and todos

 * Configuration management overhaul [Chad]
 * Fundraising flesh-out
 * HipHop deployment (deferred because Facebook has HipHopVM)
 * Unicode normalization considerations
 * Schema handling improvements (Abstracted database schema description, schema updaters to make the multiple schema variants for MySQL 4.0 and 5/UTF-8 more maintainable)
 * Parallelizable, restartable dump runner (notes; Clean up the currently fragile externaldump process, make our internal backups more reliable
 * OAI cleanup (Clean up OAI exporter and importer, add time-delay features
 * audit users of OAIPMH and check how search indexing uses it
 * Deletion queue system (notes; Human-helpful process improvements
 * Wikidata (WMDE)
 * BlameMaps (WMDE/Luca de Alfaro; see proof of concept)
 * Various HTTPS support fixes
 * Wikimedia server error page
 * Disk-backed object cache
 * Relicense our Lucene contributions [future Lucene search engineer]
 * OAuth [ChrisS]
 * related to potential API redesign (Wikia)
 * Changes to visual design to make ACW consistent with Design Toolkit [TBD]
 * Features parking
 * Queuing infrastructure work for Echo (probably using Aarons JobQueue abstraction)
 * Article creation workflow deferred
 * Gadgets 2.0?
 * Affiliation tools for WikiProjects [build on GlobalProfile]
 * Git/Gerrit
 * Have Lucene work underway upstream for 2.6 release cycle [Chad]

Archive
Previous months:
 * Roadmap/Archive for an earlier effort to create a roadmap more focused on point releases of the MediaWiki software
 * /2011/September
 * /2011/October
 * /2011/November
 * /2011/December
 * /2012/January
 * /2012/February
 * /2012/March
 * /2012/April
 * /2012/May
 * /2012/June
 * /2012/July
 * /2012/August
 * /2012/September
 * /2012/October
 * /2012/November