Roadmap

Background: In recent years, Wikimedia Foundation's engineering planning has primarily taken place in a combination of internal goals documents and public project-level documentation. The purpose of this page is to establish a practice of sharing holistic engineering goals, initially primarily with a short term horizon, as a public document with open participation. See also: Previous months:
 * Product Whitepaper for Wikimedia Foundation product priority justifications; Strategic Plan for long-term WMF goals, some of which are engineering goals
 * Roadmap/Archive for an earlier effort to create a roadmap more focused on point releases of the MediaWiki software
 * /2011/September
 * /2011/October
 * /2011/November
 * /2011/December
 * /2012/January
 * /2012/February
 * /2012/March
 * /2012/April
 * /2012/May
 * /2012/June
 * /2012/July
 * /2012/August
 * /2012/September

Item syntax:
 * Project name
 * [Qualifier keyword] Action item [Owner(s)] [Completion keyword]

Examples:
 * New unicorn engagement
 * [Needs consensus] Deploy unicorns [Brion/Tim]
 * Deploy unicorns [Brion/Tim] [DONE]

Continuous Responsibilities

 * Code review
 * RT review / ops tasks
 * Community extension review and deployment
 * Bug triages and general bug fixes
 * New extension deployment (Review queue)
 * Labs maintenance
 * Test infrastructure maintenance
 * Recruitment and encouragement of volunteers
 * Monthly report and other engineering communications

Technical Operations

 * Redis/Memcached
 * Deploy limited php-redis on the apaches, and all new sessions are being written to a redis instance in addition to memcached. [Tim/Asher]
 * Swift
 * start migrating swift backend servers to new NAS1 server
 * Evaluate Ceph [Faidon/Asher/Aaron]
 * Commons infrastructure
 * migrate ms7 (commons) to Nas1 [Done]
 * replicate Nas1 (Tampa) to Nas2 (Ashburn)
 * Parser Cache
 * build 2 more parser cache instances@ Tampa
 * Deploy Etherpad Lite [Daniel/RobH]

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * API finalization/debugging [IN PROCESS]
 * Iteration on Visual Editor for Dec release [IN PROCESS]
 * Parsoid testing framework for semantic/superficial regressions [IN PROCESS]
 * Node.js parsoid to completion [ONGOING]
 * experimentation into C/C++ version [IN PROCESS]


 * New Pages Feed (fka Page Triage) (Kaldari, Benny)
 * Descriptive statistics on NPF (revised dashboard on toolserver) [Dario] [IN PROCESS]
 * Bug fixes (no more feature development)
 * Additional socialization and public announcement/launch [DONE]


 * Echo_(Notifications) (Kaldari, Benny, Werdna, Luke Welling, Preilly, Aaron S)
 * re-deployment of prototype for Mediawiki.org [Werdna/Vibha] [IN PROCESS]
 * Continued development of prototype for Mediawiki.org [Werdna/Vibha/Kaldari/Benny] [ONGOING]
 * Selection of initial use cases [Vibha/Fabrice] [DONE]
 * UI and bugfixing of Echo [ONGOING]
 * Global user changes [JamesF] [IN PROCESS]
 * Global preferences for echo delivery [Deferred to 2013]
 * centralization and cross wiki notification delivery [Deferred to 2013]
 * (see Performance engineering for infrastructure prep work:)
 * queueing & Echo
 * Database sharding & Flow


 * Article feedback (Matthias)
 * performance/db related changes to AFTv5 queries before scaling, + sharding (see Database sharding) [Matthias] [IN PROCESS]
 * Oversight checklist (to help reduce the volume of oversight requests) [Oct 11]
 * Continued AbuseFilter work [IN PROCESS]
 * Conversion analysis: [starts Oct 11]


 * E3 (Ori, S, Luke)
 * Clicktracking/Analytics
 * Event tracking live [DONE]
 * EventTracking schema filtering [IN PROCESS]
 * Clicktracking sunset [Scheduled for 10/31]


 * Post-edit Feedback 2
 * Deployed to production [DONE]
 * Data analysis once experiment is complete [IN PROCESS]


 * Account Creation User Experience
 * Deployment of first test to production [DONE]
 * Deployment of second test to production (client-side validation) [10/24]
 * Minor UI tweaks [IN PROCESS]
 * Data analysis once experiment is complete [IN PROCESS]

Fundraising

 * Data discrepency error [IN PROCESS]

Multimedia

 * TimedMediaHandler
 * Bugfixing of player on test2 IN PROGRESS (Michael Dale)
 * Deployment of transcoding boxes
 * Create production deployment plan
 * Create new async upload API for large file uploads [Aaron]
 * Add new async upload API support to UploadWizard [Jan]
 * Multi-cache support in MediaWiki (Varnish/Squid/etc) [Aaron]

Mobile

 * Beta features
 * left nav DONE
 * ribbon [DONE]
 * new typography [DONE]
 * solr for GeoData
 * login [DONE]
 * Prod
 * left nav DONE
 * new typography [DONE]


 * promote wikipedia beta app to stable
 * mobile friendly resource loader [DONE]
 * dynamic sections [DONE]
 * Analyze WLM data to guide next upload steps
 * pickup any remaining redirects [office, chapters, ...] [IN-PROGRESS]
 * Launch Saudi [DONE]

QA

 * Created basic Regression Tests Mobile Browser Regression Tests
 * Start Implementing Watir-based browser Mobile Testing Watir-webdriver Mobile Testing
 * Continue working with Chris and Zeljko to use CI framework that the Platform QA team is setting up
 * Continue to work with the Product team to Test prior to each deployment

MediaWiki Core

 * Datacenter migration (see TechOps section above)
 * Git/Conversion (Chad/Antoine)
 * Mirror mediawiki-core to GitHub [Chad] [DONE]
 * Have last of extensions out of SVN and into Git [Done]
 * Mark extensions directory as read-only [Chad]
 * Finish replicating all repos to GitHub [IN PROGRESS]
 * Begin researching how to get pull requests back in
 * Upgrade to 2.5 [ON HOLD]
 * Swift
 * See Swift section in Ops
 * MediaWiki 1.20/Roadmap
 * Bi-weekly deployment cycle continues
 * MediaWiki 1.21/wmf1 [DONE]
 * MediaWiki 1.21/wmf2 [3/4 DONE]
 * Continuous integration
 * Integrate zuul into Jenkins pipeline [Antoine] [IN PROGRESS]
 * Move Jenkins job setup out of web interface into Git using Jenkins Job Builder [Antoine] [AFTER ZUUL]
 * Upgrade Gallium to Precise [(someone in Ops) + Antoine] [ON HOLD PENDING OPS]
 * Admin tools development
 * Deploy global AbuseFilter improvements [CSteipp] (https://gerrit.wikimedia.org/r/#/c/23509/ / https://gerrit.wikimedia.org/r/#/c/23514/ )
 * Build and deploy CentralAuth mass account locking [CSteipp]
 * Wikidata deployment
 * Code review of extensions [Sam, Tim, Chris] [DONE]
 * Need to review DataValues
 * Deploy MediaWiki version with ContentHandler branch merged in [Sam] [Done]
 * Create and deploy wikidata.org [Sam,Chad]
 * Deploy Wikibase client to test2 [Chad]
 * Site performance
 * Lua scripting [Tim]
 * Benchmarking Lua templates relative to Wikitext counterparts [Tim]
 * Wikivoyage migration [Chris, Sam]
 * Deployment to en.wikivoyage to Beta [DONE]
 * Deployment to other langages [ongoing]

QA

 * Beta labs
 * Continue refining beta labs performance [Antoine/Chris/Željko]
 * Jenkins integration with gerrit is ongoing and is critical to consistent beta labs deployment and also critical to effective browser tests.
 * Begin work on Watir-based browser testing framework
 * Create at least one useful end-to-end test for important Wikipedia feature, with automatic execution (Jenkins-driven) and results reporting [Chris/Željko]
 * Create a list of targets for browser testing according to priority
 * Continue integrating Mobile QA with overall QA/testing activities

Analytics
See also: Analytics/Roadmap

Kraken

 * Set up Maven / Somatype Artifact Repository (continuous integration) [dsc]
 * WMF Maven parent pom [dsc]
 * Puppetize Kraken [otto] (Ongoing)
 * Set up JMX monitoring -- needs to be our LAN [otto + dsc]
 * Hadoop Ganglia Monitoring [otto] (Oct)
 * Get Storm set up [dsc + otto]
 * Start work on ETL topology [dsc]
 * Hardware reinstallation -- Depends on Ops [otto]
 * Get to consensus with Ops regarding logging of the firehose [dsc + otto]
 * Research needed: test running cli JVM producers does not cause extra load [otto]

Limn

 * Refactor charting to use d3 [dan + dsc]
 * Feature Parity with Dygraphs (plus bugfixes, etc) (Oct)
 * Bugfixes (like Save-As, UI Error Notifications, ...) [dan] (Oct)
 * Improve Limn wiki, docs, & guides (esp those shameful screenshots) [dan] (Oct)
 * UI support for remote datasets via proxy [dsc + dan] (Oct)
 * Deploy reportcard / gerrit-stats on stat1001 (aka, "the Debian packaging discussion") [otto + dsc]
 * Support the Global Dev dashboard [evan] (Ongoing)
 * Support the Gerrit Stats dashboard [diederik] (Ongoing)

Legacy Log Collection

 * Add support for new domain names in webstatscollector (blog, etc) [diederik]
 * udp2log filters
 * Update filters for Wikipedia Zero [otto] (Ongoing)
 * Filter by X-Carrier headers. [otto + asher + diederik]
 * udp-filter to filter by http status. [otto]

WikiStats

 * Reduce backlog regarding Wikistats traffic (squid etc) scripts [stefan]
 * Repair data errors in wikistats, and add process for checking data integrity [ezachte] (Sept)
 * Make wikistats more robust (MoM validations) [ezachte]
 * Add Blackbox testing to WikiStats [diederik + ezachte]

Infrastructure

 * Access/support requests for stat1, stat1001 [otto] (Ongoing)
 * Migrate Reportcard off Labs onto stat1001 -- reportcard.wikimedia.org [otto + dsc] (Oct)
 * Maintenance of oxygen/emery/locke [otto] (Ongoing)

Data Releases

 * Publish Monthly Report Card -- deal with monthly data processing irregularities, perform correction/validation [ezachte + diederik + dsc] (Ongoing)
 * Create Data Release Practices Task Force [diederik]
 * Start pushing datasets to AWS [diederik]
 * Finalize scripts to massively compact dammit.lt data [erik]
 * Blogpost about what awesome stuff you can do with this [diederik + ?]

Engineering Community Team

 * Bug Wrangling [Andre + volunteers] -- ongoing
 * Updating BZ maintenance documentation [Andre] -- in progress
 * Followup from events [Sumana] -- ongoing
 * Followup on Grace Hopper
 * Tech Ambassadors Network [Guillaume] -- ongoing on meta
 * Volunteer product management program [Guillaume] -- being integrated into Tech/Ambassadors
 * Misc Engineering community proposals
 * 20% time [Sumana] -- ongoing - target 26 Oct
 * CoC [Sumana] -- in process

i18n/L10n

 * Internationalization/Localization
 * Universal Language Selector
 * Version 1 development [IN PROGRESS]
 * User Experience / UI testing [ONGOING]
 * Technical documentation [IN PROGRESS]
 * Project Lassi: Translate UI/UX Improvements
 * User Experience / UI testing Round 1 [DONE]
 * UI design finalization for development [IN PROGRESS]
 * Prototype development [IN PROGRESS]
 * Translate API
 * API definition [On Hold]
 * Central Notice-Translate integration
 * Code changes DONE
 * Code review/testing: The i18n team has put this integration project on hold until availability from fundraising engineers to complete code review for deployment. ON HOLD
 * Deployment - scheduled for next sprint SCHEDULED
 * Web Fonts
 * Adding languages [ONGOING]
 * Narayam
 * Adding input methods for other languages [ONGOING]
 * Project Milkshake
 * Design specification, API specification, Documentation [ONGOING]
 * jQuery.i18n Development [IN PROGRESS]
 * jQuery.uls Development [IN PROGRESS]
 * jQuery.ime Development [IN PROGRESS]
 * jQuery.webfonts Development [IN PROGRESS]
 * Technical documentation [DONE]
 * Language search
 * Initial analysis [DONE]
 * Prototype initial concept [IN PROGRESS]
 * Integration into ULS [IN PROGRESS]
 * Standards participation
 * Comments on ARB Specification [DONE]
 * Language Summit
 * Preparation for November 6-7 Language Summit [IN PROGRESS]
 * Language Team Community Meetup
 * Language Team Pune Meetup Nov 4 [IN PROGRESS]
 * Language Team Bangalore Meetup Nov 9 [IN PROGRESS]
 * India DevCamp
 * Preparation of Topics for DevCamp [IN PROGRESS]
 * Registration now open [IN PROGRESS]

Features

 * VisualEditor/Parsoid (Trevor, Roan, Rob, Gabriel, Subbu, MarkTraceur, + Wikia)
 * API finalization/debugging [IN PROCESS]
 * Iteration on Visual Editor for Dec release [IN PROCESS]
 * Parsoid testing framework for semantic/superficial regressions [IN PROCESS]
 * Node.js parsoid to completion [ONGOING]
 * experimentation into C/C++ version [IN PROCESS]


 * New Pages Feed (fka Page Triage) (Kaldari, Benny)
 * Descriptive statistics on NPF (revised dashboard on toolserver) [Dario] [IN PROCESS]
 * Bug fixes (no more feature development)


 * Echo_(Notifications) (Kaldari, Benny, Werdna, Luke Welling, Preilly, Aaron S)
 * Continued development of prototype for Mediawiki.org [Werdna/Vibha/Kaldari/Benny] [ONGOING]
 * UI and bugfixing of Echo [ONGOING]
 * Global user changes [JamesF] [IN PROCESS]
 * (see Performance engineering for infrastructure prep work:)
 * queueing & Echo
 * Database sharding & Flow


 * Article feedback (Matthias)
 * performance/db related changes to AFTv5 queries before scaling, + sharding (see Database sharding) [Matthias] [IN PROCESS]
 * Post-edit Feedback 2
 * Data analysis once experiment is complete [IN PROCESS]


 * Account Creation User Experience
 * Minor UI tweaks [IN PROCESS]
 * Data analysis once experiment is complete [IN PROCESS]

Fundraising

 * Data discrepancy error [IN PROCESS]
 * Start the Fundraiser in early November, no date set yet, tentative target date is Nov 12

Multimedia

 * TimedMediaHandler
 * Bugfixing of player on test2 IN PROGRESS (Michael Dale)
 * Deployment of transcoding boxes
 * Create production deployment plan
 * Create new async upload API for large file uploads [Aaron]
 * Add new async upload API support to UploadWizard [Jan]
 * Multi-cache support in MediaWiki (Varnish/Squid/etc) [Aaron]

MediaWiki Core

 * Datacenter migration (see TechOps section above)
 * Git/Conversion (Chad/Antoine)
 * Have Lucene work underway upstream for 2.6 release cycle [Chad]
 * Have last of extensions out of SVN and into Git [IN PROGRESS]
 * Finish replicating all repos to GitHub [ON HOLD PENDING 2.5]
 * Begin researching how to get pull requests back in
 * Replace gitweb with GitBlit [Chad]
 * Swift
 * See Swift section in Ops
 * MediaWiki 1.20/Roadmap
 * Bi-weekly deployment cycle continues
 * MediaWiki 1.21/wmf3
 * MediaWiki 1.21/wmf4
 * Continuous integration
 * Admin tools development [Chris]
 * Build and deploy Global RenameUser extension [CSteipp]
 * Wikidata deployment
 * Site performance
 * Lua scripting [Tim]
 * Wikivoyage migration [Chris, Sam]

QA

 * Beta labs
 * Continue refining beta labs performance [Antoine/Chris/Željko]
 * Automated testing
 * Feature testing
 * Mobile
 * Visual Editor
 * E2

Analytics
See also: Analytics/Roadmap

Kraken

 * First pixel/firehose prototype: udp2log -> product code topic filters -> Kafka -> hadoop [otto] (Nov 9)
 * Product Code wiki & email stakeholders [dsc] (Nov)
 * Reinstall Dells, Fix Cisco machines [otto] (Nov)
 * Puppetize Kraken [otto] (Ongoing)
 * Monitoring [otto] (Nov)
 * Ganglia Monitoring [otto] (Nov)
 * Set up JMX monitoring -- needs to be our LAN [otto + dsc] (Nov)
 * Set up Maven / Somatype Artifact Repository (continuous integration) [dsc] (Nov)
 * WMF Maven parent pom [dsc] (Oct) [DONE]
 * Walkthrough to get up and running with Maven in Eclipse [dsc] (Nov)
 * Storm pom [dsc] (Nov)

Engineering Community Team

 * Volunteer coordination and outreach
 * Event planning & followup [Sumana, Technical Contributor Coordinator]
 * Online or in-person QA trainings for the community [Technical Contributor Coordinator]
 * Community testing events [Technical Contributor Coordinator]
 * UCOSP idea-wrangling [Sumana] -- ongoing
 * Bug Wrangling [Andre + volunteers]
 * Updating BZ maintenance documentation [Andre]
 * Summer of Code 2012
 * Postmortem [Sumana]
 * Tech Ambassadors Network [Guillaume]
 * Volunteer product management program [Guillaume]

Technical Operations

 * Labs
 * Push OpenStackManager changes to show SSH fingerprints for instances [Ryan Lane] (not a high-priority item, will probably be December 2012 or later)
 * Add DNS support to Nova (in essex, but we must migrate to using it)
 * Add Gluster support to Nova (Gluster plugin is mostly written but we don't really have plugin support until folsom, so will happen in Q3 when we are ready to upgrade to Folsom)

Features

 * Echo_(Notifications)
 * Global preferences for echo delivery [Deferred to 2013]
 * centralization and cross wiki notification delivery [Deferred to 2013]

Parking lot and todos

 * Install Lua (Scribunto) on other Wikimedia wikis (probably sometime in 2013; see Goals)
 * Category collation
 * Patch waiting on Liangent for more work for full deployment
 * Queuing infrastructure work for Flow (who added this?)
 * Configuration management overhaul [Chad]
 * Evaluate WikiFactory from Wikia
 * Extension management -- depends on config mgmt revamp
 * graphical UI on top of improved backend for configuration management
 * Details of git migration (extension mgmt, permissions, code review tool, Gerrit-Jenkins integration, etc.)
 * More work on product backlogs for features and mobile
 * Ops section
 * Fundraising flesh-out
 * cf Wikimedia Platform Engineering and merge as appropriate
 * Namespace hacks for development of simple language wikipedias in languages other than English without server, database, or major wiki installation provisioning
 * Most popular related articles
 * HipHop deployment (deferred because Facebook has a different shiny new thing)
 * Extension:Configure (merge with config management?)
 * Unicode normalization considerations
 * Schema handling improvements (Abstracted database schema description, schema updaters to make the multiple schema variants for MySQL 4.0 and 5/UTF-8 more maintainable)
 * Parallelizable, restartable dump runner (notes; Clean up the currently fragile externaldump process, make our internal backups more reliable
 * OAI cleanup (Clean up OAI exporter and importer, add time-delay features
 * audit users of OAIPMH and check how search indexing uses it
 * Deletion queue system (notes; Human-helpful process improvements
 * Integrated heuristic checks, improved patrolling (human-helpful process improvements)
 * On-wiki namespace manager (Special:Namespaces) (original wikidata branch specs and see notes)
 * Wikidata (WMDE)
 * BlameMaps (WMDE)
 * Various HTTPS support fixes
 * Wikimedia server error page
 * Wikimedia operations history and architecture document
 * Disk-backed object cache
 * DB sharding
 * Help features
 * Mentoring features
 * Other real-time features, chat
 * Affiliation tools for WikiProjects [build on GlobalProfile]
 * Gadgets 2.0?
 * Interwiki transclusion
 * WMF cannot commit the time to merge this.
 * Replicated session handling [Tim Starling]
 * Relicense our Lucene contributions [future Lucene search engineer]
 * OAuth [ChrisS]
 * related to potential API redesign (Wikia)
 * Article Creation Workflow: Postponed
 * Changes to visual design to make ACW consistent with Design Toolkit [TBD]