Roadmap

Background: In recent years, Wikimedia Foundation's engineering planning has primarily taken place in a combination of internal goals documents and public project-level documentation. The purpose of this page is to establish a practice of sharing holistic engineering goals, initially primarily with a short term horizon, as a public document with open participation. See also: Previous months:
 * Product Whitepaper for Wikimedia Foundation product priority justifications; Strategic Plan for long-term WMF goals, some of which are engineering goals
 * Roadmap/Archive for an earlier effort to create a roadmap more focused on point releases of the MediaWiki software
 * /2011/September
 * /2011/October
 * /2011/November
 * /2011/December
 * /2012/January
 * /2012/February
 * /2012/March
 * /2012/April
 * /2012/May
 * /2012/June
 * /2012/July
 * /2012/August
 * /2012/September

Item syntax:
 * Project name
 * [Qualifier keyword] Action item [Owner(s)] [Completion keyword]

Examples:
 * New unicorn engagement
 * [Needs consensus] Deploy unicorns [Brion/Tim]
 * Deploy unicorns [Brion/Tim] [DONE]

Continuous Responsibilities

 * Code review
 * RT review / ops tasks
 * Community extension review and deployment
 * Bug triages and general bug fixes
 * New extension deployment (Review queue)
 * Labs maintenance
 * Test infrastructure maintenance
 * Recruitment and encouragement of volunteers
 * Monthly report and other engineering communications

Technical Operations

 * Deploy Varnish on upload@esams to replace Squid [done]
 * Full Memcached deployment@Tampa [Done]
 * Swift upgrade from 1.5 to 1.74 [Faidon/Ariel] [partial]
 * Ceph evaluation [Asher / Mark / Faidon] [on-going]
 * Multiwrite thumbnails to nas01 [Aaron/Faidon/Mark] [done]
 * Upgrade MW AppServers/ImageScalers to Precise [done]
 * Complete Fundraising infrastructure buildout@Eqiad and lockdown [done]
 * Complete Wikivoyage infrastructure buildout and deploy [Daniel/Asher] [Done]
 * Provision 2 extra parser cache servers (pc2, pc3) @ Tampa [done]

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * API finalization/debugging [IN PROCESS]
 * Iteration on Visual Editor for Dec release [IN PROCESS]
 * Parsoid testing framework for semantic/superficial regressions [IN PROCESS]
 * Node.js parsoid to completion [ONGOING]


 * New Pages Feed (fka Page Triage) (Kaldari, Benny, Nischay)
 * Descriptive statistics on NPF (revised dashboard on toolserver) [Dario] [IN PROCESS]
 * Bug fixes (no more feature development) [ONGOING]


 * Echo_(Notifications) (Kaldari, Benny, Werdna, Luke Welling, Preilly, Aaron S)
 * Continued development of prototype for Mediawiki.org [Werdna/Vibha/Kaldari/Benny] [ONGOING] (note: prototype has been reverted on Mediawik.org. Redeployment is pending further feature development)
 * First community feedback via IRC Office Hours (Nov 14) [DONE]
 * UI and bugfixing of Echo [ONGOING]
 * Global user changes [JamesF] [IN PROCESS]
 * (see Performance engineering for infrastructure prep work:)
 * queueing & Echo
 * Database sharding & Flow


 * Article feedback (Matthias)
 * performance/db related changes to AFTv5 queries before scaling, + sharding (see Database sharding) [Matthias] [IN PROCESS]


 * E3 (Ori, S, Luke)
 * Clicktracking/Analytics
 * Event tracking live [DONE]
 * EventTracking schema filtering [DONE]
 * Clicktracking sunset [DONE]
 * Funnel tracking of Community Portal [ONGOING]


 * Post-edit Feedback 2
 * Data analysis once experiment is complete [DUE 10/31]
 * Continued productization [DONE]


 * Account Creation User Experience
 * Minor UI tweaks (ACUX 2, no benefits) [DONE]
 * Real-time validation test (ACUX 3)[IN PROCESS]
 * Deploy ACUX 3 to 100% on English Wikipedia during productization "[Nov 29]''
 * Productization of ACUX [IN ANALYSIS]
 * Data analysis once experiment is complete [IN PROCESS]


 * Onboarding
 * Develop wireframes and workflow [DONE]
 * Work with SuggestBot author to develop list of tasks [IN PROCESS]
 * Extensionize guided tour [IN PROCESS]
 * Deploy first version of onboarding to production [Dec 6 target]


 * MicroDesign
 * anon user template [DONE]
 * other edit window templates [IN PROCESS]
 * (see Agora)

Fundraising

 * Data discrepancy error [DONE]
 * Start the Fundraiser in early November [DONE-NOV 26]

Mobile

 * DevCamp [DONE]
 * Mobile Web
 * Beta editing [DONE]
 * Beta watchlist (add and view) [DONE]
 * Beta random [DONE]
 * Better table formatting [DONE]
 * Apps
 * iOS Bugfixes
 * landscape [DONE]
 * Release to market
 * Android
 * Offline issue
 * Solr for GeoData
 * CR [DONE]

MediaWiki Core

 * Datacenter migration (see TechOps section above)
 * Git/Conversion (Chad)
 * LDAP fixes for our Gerrit use (current patch) [Chad]
 * Usability improvements for Gerrit (collaboration at Gerrit Summit) [Chad]
 * Upgrade to 2.6 [Chad]
 * Have Lucene work underway upstream for 2.6 release cycle [Chad]
 * Replace gitweb with GitBlit [Chad]
 * Swift
 * Copying thumbnail files from Swift into NFS [Aaron]
 * MediaWiki 1.20/Roadmap
 * Bi-weekly deployment cycle continues
 * MediaWiki 1.21/wmf4 [DONE]
 * MediaWiki 1.21/wmf5 ongoing
 * Continuous integration
 * Integrate zuul into Jenkins pipeline [Antoine] -- [DONE]
 * Move Jenkins job setup out of web interface into Git using Jenkins Job Builder [Antoine] [AFTER ZUUL]
 * Admin tools development [Chris]
 * Build and deploy Global RenameUser extension [CSteipp]
 * Wikidata deployment
 * Make ULS support for logged-in users only [Wikidata team]
 * Deploy Wikibase client to test2 [Chad]
 * Site performance
 * One-off template profiling [Tim] [DONE]
 * Lua scripting [Tim, Chad, Brad]
 * Deploy TemplateSandbox [Brad] [ongoing. deployed to test2, mediawiki.org]
 * Shared Lua repo [Chad]
 * Additional APIs for multi-lingual support [Tim] [DONE]
 * Detailed deployment plan [TBD]
 * Wikivoyage migration [Chris, Sam]
 * Soft launch -- DONE
 * Public launch -- DONE
 * Search
 * Added Timeouts to lsearchd [Tim]

Multimedia

 * TimedMediaHandler
 * Bugfixing urgent deployment related issues DONE (Michael Dale)
 * Deployment of transcoding boxes [DONE]
 * Create production deployment plan [DONE]
 * Create new async upload API for large file uploads [Aaron] [ongoing]
 * Add new async upload API support to UploadWizard [Jan] [DONE]
 * Multi-cache support in MediaWiki (Varnish/Squid/etc) [Aaron] [DONE]

QA

 * Beta labs
 * Continue refining beta labs performance [Antoine/Chris/Željko]
 * AFTv5 and E3 hosted reliably on beta, prototype host is decommissioned [DONE]
 * PageTriage code is hosted reliably on beta, state of db contents may not be up to date [IN PROCESS] [Antoine]
 * Automated testing
 * Ported reporting frameworks from raw RSpec to Cucumber (see below) [ONGOING]
 * Tests are running under Jenkins in Cloudbees against test2 and beta. [DONE]
 * Make Cloudbees Jenkins slave to WMF Jenkins (Netherlands hackathon did not get to this)
 * Fixing glitches with file handling in Jenkins (works locally)
 * Creating new tests for AFTv5 [ONGOING]
 * Created [|test backlog page] . Cucumber format and backlog should enable wider participation in ATDD (Acceptance Test Driven Development) [ONGOING]
 * Coverage goals: regression tests from backlog page, extensions managed on beta. [ONGOING]
 * Feature testing
 * Mobile
 * Create first Mobile browser tests [NOT DONE]
 * Visual Editor
 * VE user interface is being re-written in November, following stability issues for beta support and UI testing [NOT DONE]
 * E2
 * Significant AFTv5 testing on hold pending sharding, creating browser regression tests meanwhile [ONGOING]
 * PageTriage development moving slowly, focusing on support in beta labs [ONGOING]

Analytics
See also: Analytics/Roadmap

Kraken

 * Pixel Service
 * First Prototype: udp2log -> product code topic filters -> Kafka -> hadoop [otto] (Nov 9) [DONE]
 * Pixel Service documentation [dsc] (Nov) [DONE]
 * Work out Event Data conventions for proxied fields / normal web request components [dsc, e3] (Nov) [DONE]
 * Product Code wiki [dsc] (Nov) [DONE]
 * Email stakeholders and gather feedback [dsc] (Dec)
 * Reinstall Dells [otto] (Nov) [DONE]
 * Fix Cisco machines (an02, an07) [otto] (Nov) (Awaits ops help)
 * Puppetize Kraken [otto] (Ongoing)
 * Monitoring [otto] (Nov)
 * Ganglia Monitoring [otto] (Nov) [DONE]
 * Set up JMX monitoring -- needs to be our LAN [otto + dsc] (Nov)
 * Solution Research page [dsc] (Nov) [DONE]
 * Core Jobs
 * Data Formats wiki page [dsc + diederik] (Nov) [DONE]
 * Avro Schemas for Request & Event Data records [dsc + diederik] (Dec) [DONE]
 * Set up full pixel/firehose (w/o ETL) via  [otto] (Dec) (Ready to go, holding off on flipping switch)
 * Figure out varnishncsa log format for event stream [otto + dsc] (Nov) [DONE]
 * Get to consensus with Ops regarding logging of the firehose [dsc + otto] [DONE]
 * Research needed: test running cli JVM producers uses acceptable resources [otto] (Oct) [not doing]

Limn

 * Refactor charting to use d3 [dan + dsc]
 * Feature Parity with Dygraphs (plus bugfixes, etc) (Dec) [DONE]
 * Support the Global Dev dashboard [evan] (Ongoing)
 * Support the Gerrit Stats dashboard [diederik] (Ongoing)

Legacy Log Collection

 * Add support for new domain names in webstatscollector (blog, etc) [diederik] (Sept) [Status: Made a Baby Instead???]
 * udp2log filters
 * Update filters for Wikipedia Zero [otto] (Ongoing)
 * Filter by X-Carrier headers. [otto + asher + diederik] (Oct) [Status: At Least We Think It's a Baby?]
 * udp-filter to filter by http status. [otto] (Oct) [Status: Maybe It's a T-Rex!!!]

WikiStats
(Stale -- will consult Stefan)
 * Reduce backlog regarding Wikistats traffic (squid etc) scripts [stefan] (Oct)
 * Repair data errors in wikistats, and add process for checking data integrity [ezachte] (Sept)
 * Make wikistats more robust (MoM validations) [ezachte] (Oct)
 * Add Blackbox testing to WikiStats [diederik + ezachte] (Oct)

Infrastructure

 * Access/support requests for stat1, stat1001 [otto] (Ongoing)
 * Maintenance of oxygen/emery/locke [otto] (Ongoing)
 * Kraken
 * Reinstall Dells [otto] (Nov) [DONE]
 * Fix Cisco machines (an02, an07) [otto] (Nov) (Awaits ops help)

Data Releases

 * Start pushing datasets to AWS [diederik] (Nov) [DONE]
 * Blogpost about what awesome stuff you can do with this [diederik + ?] (Oct) [Status: BABYPOCALYPSE 2k12]
 * Finalize scripts to massively compact dammit.lt data [erik] (Oct) [???]
 * Create Data Release Practices Task Force [diederik] (Sept) [Status: Baaaaaby!]
 * Publish Monthly Report Card -- deal with monthly data processing irregularities, perform correction/validation [ezachte + diederik + dsc] (Ongoing)

Engineering Community Team

 * Volunteer coordination and outreach
 * Event planning & followup [Sumana, Quim]
 * Planning online or in-person QA trainings for the community [Quim]
 * Planning community testing events [Quim]
 * Bug Wrangling [Andre + volunteers]
 * Updating BZ maintenance documentation [Andre] -- ongoing
 * Improving workflow for highest-priority bugs escalation -- ongoing [Andre]
 * Summer of Code 2012
 * Postmortem [Sumana] - [done]
 * Pushing on merges -- ongoing [Sumana]
 * Outreach Program for Women - ongoing [Quim, Sumana]
 * Tech Ambassadors Network [Guillaume] -- ongoing on meta
 * Volunteer product management program [Guillaume] -- being integrated into Tech/Ambassadors
 * Misc Engineering community proposals
 * 20% time/LevelUp [Sumana] -- ongoing - socialized internally, soon to launch for real
 * CoC [Sumana] -- in process - target 30 Nov

Technical Operations

 * Swift @ Tampa upgrade from 1.5 to 1.74 [Faidon/Ariel]
 * Ceph evaluation [Asher / Mark / Faidon] [on-going]
 * Copy thumbnails from Swift  to nas01 [Aaron/Faidon/Mark]
 * Productionize 2 extra parser cache servers (pc2, pc3) @ Tampa [Asher]
 * Deploy Varnish for upload@ESAM [Mark]
 * Swift Servers swap out (C2100 to R720XD) @ Tampa [Chris, Ariel]
 * Provision Memcached servers @ Eqiad [Peter]
 * Provision parser cache servers @ eqiad [RobH, Peter, Asher]


 * Labs
 * Push OpenStackManager changes to show SSH fingerprints for instances [Ryan Lane] (not a high-priority item, will probably be December 2012 or  later)
 * Add DNS support to Nova (in essex, but we must migrate to using it)
 * Add Gluster support to Nova (Gluster plugin is mostly written but we don't really have plugin support until folsom, so will happen in Q3 when  we are ready to upgrade to Folsom)
 * Deploy Etherpad Lite [Daniel/RobH]

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * Iteration on Visual Editor for Dec release [IN PROCESS]
 * Parsoid testing framework for semantic/superficial regressions [IN PROCESS]
 * Node.js parsoid to completion [ONGOING]


 * New Pages Feed (fka Page Triage) (Kaldari, Benny, Nischay)
 * Descriptive statistics on NPF (revised dashboard on toolserver) [Dario] [IN PROCESS]
 * Bug fixes (no more feature development) [ONGOING]


 * Echo_(Notifications) (Kaldari, Benny, Werdna, Luke Welling, Preilly, Aaron S)
 * Continued development of prototype for Mediawiki.org [Werdna/Vibha/Kaldari/Benny] [ONGOING]
 * UI and bugfixing of Echo [ONGOING]
 * Global user changes [JamesF] [IN PROCESS]
 * (see Performance engineering for infrastructure prep work:)
 * queueing & Echo
 * Database sharding & Flow


 * [Flow] (Werdna, JamesF, Brandon, Matthias)
 * Kickoff
 * reach out to Wikia [Terry,JamesF]


 * Article feedback (Matthias)
 * pilot deploy of AFTv5 to dewiki [DONE]
 * performance/db related changes to AFTv5 queries before scaling, + sharding (see Database sharding) [Matthias] [IN PROCESS]
 * Other potential, minor changes prior to full English Wikipedia deployment [IN PROCESS]


 * E3 (Ori, S, Luke, MattF)
 * Clicktracking/Analytics
 * Datamodel [IN PROCESS]
 * Funnel tracking of Community Portal [ONGOING]


 * Account Creation User Experience
 * Productization of ACUX [IN PROCESS]
 * Data analysis once experiment is complete [IN PROCESS]


 * Onboarding
 * Extensionize guided tour [IN PROCESS]
 * Deploy first version of onboarding to production [Dec 6]
 * Continue working with SuggestBot author to develop list of tasks [IN PROCESS]


 * MicroDesign
 * other edit window templates [IN PROCESS]
 * (see Agora)


 * List of other projects on radar
 * ResourceLoader2
 * Article creation workflow
 * LiquidThreads 3.0

Fundraising (Katie, Adam, MattW, Peter)

 * Fundraiser - 5 countries [IN PROGRESS]

Mobile

 * Mobile Web
 * Beta photo upload
 * User Testing of Beta Editing, next steps TBD
 * Features under consideration for next sprint: Account Creation, Reader-friendly Watchlist, Last Modified, History, AFT (?)

MediaWiki Core

 * Score deployment to test2 [Tim] - deployment to test2 done, further deployment waiting on OggHandler-TMH dependency fix

Analytics
See also: Analytics/Roadmap

Kraken

 * Set up full pixel/firehose (w/o ETL) via  [otto] (Dec) (Ready to go, holding off on flipping switch)
 * Core Jobs: aggregation, bucketing, indices [dsc, DvL, erosen] (Dec/Ongoing)
 * Product Codes: Email stakeholders and gather feedback [dsc] (Dec)
 * Set up Maven / Somatype Artifact Repository (continuous integration) [dsc] (Dec)
 * WMF Maven parent pom [dsc] (Oct) [DONE]
 * Walkthrough to get up and running with Maven in Eclipse [dsc] (Dec)
 * Storm pom [dsc] (Dec)
 * Storm ETL [dsc + otto] (Dec)
 * Get Storm set up [dsc + otto] (Dec)
 * Storm ETL bolts for (some of) GeoIP, Anonymization, HDFS Import, Kafka Checkpointing [dsc] (Dec)
 * Consume some fraction (1:1000, 1:10000?) of web access logs [otto] (Dec)
 * Consume 1:1 web access logs into HDFS with ETL + Bucketing + Tagging [otto] (Jan)

Limn
Major release (0.8) planned for 5 Dec (prior to Metrics Meeting on 6 Dec)


 * Bugfixes (like Save-As, UI Error Notifications, ...) [dan] (Dec)
 * Mirror GitHub to Gerrit [dsc] (Jan)
 * Improve Limn wiki, docs, & guides (esp those shameful screenshots) [dan] (Jan)
 * Coke ( for Coco) task to create symlinks into   from an existing data repository (such as, say, analytics/reportcard/data) [dsc] (Jan)
 * Coke task to download and setup dummy testing data for ease of development [dsc] (Jan)
 * UI support for remote datasets via proxy [dsc + dan] (Dec)
 * Migrate Dario's dashboards to Limn [dsc] (Dec)
 * Deploy reportcard / gerrit-stats on stat1001 (aka, "the Debian packaging discussion") [otto + dsc] (Dec)

Infrastructure

 * Access/support requests for stat1, stat1001 [otto] (Ongoing)
 * Maintenance of oxygen/emery/locke [otto] (Ongoing)
 * Migrate Reportcard off Labs onto stat1001 -- reportcard.wikimedia.org [otto + dsc] (Dec)

Data Releases

 * Blogpost about what awesome stuff you can do with AWS Datasets [diederik + ?] (Oct) [Status: BABYMAKIN]
 * Finalize scripts to massively compact dammit.lt data [erik] (Oct!!??1) [???]
 * Create Data Release Practices Task Force [diederik] (Sept!1?) [Status: BABYMAKIN]
 * Publish Monthly Report Card -- deal with monthly data processing irregularities, perform correction/validation [ezachte + diederik + dsc] (Ongoing)

Engineering Community Team

 * Volunteer coordination and outreach
 * Event planning & followup [Sumana, Quim]
 * Bangalore followup
 * Grace Hopper followup
 * OpenStreetMap and 2013 Developers Meeting meetings
 * Planning online or in-person QA trainings for the community [Quim]
 * Planning community testing events [Quim]
 * Bug Wrangling [Andre + volunteers]
 * Updating BZ maintenance documentation [Andre]
 * Outreach Program for Women - ongoing [Quim, Sumana, Andre]
 * Tech Ambassadors Network [Guillaume] -- ongoing on meta
 * Review of consultation results & creation of new experiment - target Dec 7
 * Volunteer product management program [Sumana]
 * Misc Engineering community proposals' followup [Sumana]
 * LevelUp, CoC

Technical Operations

 * Datacenter migration (eqiad)
 * Updating the scap scripts to deal with multi-datacenter

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * experimentation into C/C++ version [DEFERRED UNTIL JAN 2013]


 * Echo_(Notifications)
 * Global preferences for echo delivery [Deferred to 2013]
 * centralization and cross wiki notification delivery [Deferred to 2013]


 * Article feedback (Matthias)
 * Full deployment to English Wikipedia [2013]

Parking lot and todos

 * Install Lua (Scribunto) on other Wikimedia wikis (probably sometime in 2013; see Goals)
 * Category collation
 * Patch waiting on Liangent for more work for full deployment
 * Queuing infrastructure work for Flow
 * Configuration management overhaul [Chad]
 * Evaluate WikiFactory from Wikia
 * Extension management -- depends on config mgmt revamp
 * graphical UI on top of improved backend for configuration management
 * Fundraising flesh-out
 * cf Wikimedia Platform Engineering and merge as appropriate
 * Namespace hacks for development of simple language wikipedias in languages other than English without server, database, or major wiki installation provisioning
 * HipHop deployment (deferred because Facebook has a different shiny new thing)
 * Extension:Configure (merge with config management?)
 * Unicode normalization considerations
 * Schema handling improvements (Abstracted database schema description, schema updaters to make the multiple schema variants for MySQL 4.0 and 5/UTF-8 more maintainable)
 * Parallelizable, restartable dump runner (notes; Clean up the currently fragile externaldump process, make our internal backups more reliable
 * OAI cleanup (Clean up OAI exporter and importer, add time-delay features
 * audit users of OAIPMH and check how search indexing uses it
 * Deletion queue system (notes; Human-helpful process improvements
 * Wikidata (WMDE)
 * BlameMaps (WMDE/Luca de Alfaro; see proof of concept)
 * Various HTTPS support fixes
 * Wikimedia server error page
 * Wikimedia operations history and architecture document
 * Disk-backed object cache
 * DB sharding
 * Affiliation tools for WikiProjects [build on GlobalProfile]
 * Gadgets 2.0?
 * Interwiki transclusion
 * WMF cannot commit the time to merge this.
 * Replicated session handling [Tim Starling]
 * Relicense our Lucene contributions [future Lucene search engineer]
 * OAuth [ChrisS]
 * related to potential API redesign (Wikia)
 * Article Creation Workflow: Postponed
 * Changes to visual design to make ACW consistent with Design Toolkit [TBD]