Wikimedia Engineering/Service and REST API team

One of outcomes of the architecture summit in January was a consensus that we should move towards a service-oriented architecture. Major reasons for this are:


 * enabling new engineers to be more productive by letting them concentrate on a part of the stack
 * improved reliability and security through fault and security context isolation
 * ease of testing, debugging, monitoring and scaling
 * supporting new uses of our data by rich clients and mobile through a well-structured and efficient public content API

A storage service and a REST content API were seen as particularly good candidates for next steps in this direction. There has been some preliminary work on the Rashomon revision storage service by the Parsoid team and a generic PHP service interface with support for parallel operations by Aaron.

This has happened as side projects, with other team responsibilities taking priority. We need to do more if we really want to reap the benefits of a service-oriented architecture.

I (Gabriel Wicke) believe that we should set up a small team to move this forward. This team will


 * guide existing work to think in terms of independent services and versioned APIs,
 * help to develop internal and external REST APIs (ex: Flow),
 * help to set up automated tests against those APIs,
 * identify and implement general backend services in support of other teams (ex: storage service), and
 * support packaging for internal and third party distribution.

In the short term, this can help other teams by removing some of their workload, avoiding duplicate work and helping with API implementations.

In the longer term, it will
 * build up a consistent REST content API to enable mobile and innovative feature projects,
 * provide back-end services like the Rashomon storage service, and
 * improve testing and thus the ability to deploy more often without breaking things.

Ultimately, it should help us make real progress towards a service-oriented architecture with strong APIs. It will hopefully also result in improved developer productivity, testability, security and agility as discussed in the SOA RFC.

Misc API end points / services

 * Page metadata API for rendering of red links and other bits in Flow, VE, Mobile and eventually regular page views
 * Citation expansion service API for VE: expand a URL to a full citation
 * HTML load / save API: generalized version of the private VisualEditor API and based on the Parsoid REST API, to be used by VE, Kiwix, external editors, bots, gadgets (see current Parsoid users despite missing save API)
 * Maintain PDF render service
 * Maintain Math render service (Mathoid)
 * Possibly develop new geodata APIs as needed

API design and prototyping support

 * Flow REST API for use by rich front-end, mobile
 * possibly help to refactor Flow backend to take more advantage of existing infrastructure, improve performance

Testing

 * Drive automated testing against APIs
 * Enables test-driven development and rapid iterations; avoids annoying users with bugs

Documentation

 * Help establish best practices in declarative API documentation using tools like swagger
 * Enable third party app, gadget and bot development

Packaging

 * Package services for practical third-party and internal use

Rashomon revision storage service

 * improves VisualEditor performance on both load & save and saves bandwidth for Mobile by replacing Varnish with static storage and removing metadata from the HTML
 * ability to use this for regular page views autumn 2014
 * improved page view performance for editors
 * significantly reduced load on PHP cluster (HW cost and energy savings)
 * seamless and fast switching from page view to VE (edit metadata loaded on demand, HTML used directly)
 * support for cross-datacenter replication, compression and even load distribution across storage cluster

Generalization to Key-Value storage and counters

 * Simplified storage abstraction for extensions and other teams
 * Abstracts issues of replication, compression and load balancing

2 engineers: no new reqs
Main differences vs. 4 engineers:
 * APIs needed for Flow, VE, Mobile and Parsoid HTML in normal page views finished later
 * Causes work-around efforts as currently already happening for red links
 * Performance improvements delayed
 * Less progress on testing, documentation
 * No bandwidth for new APIs like geodata
 * No generalization of Rashomon to other storage types

Misc API end points / services

 * Page metadata API for rendering of red links and other bits in Flow, VE, Mobile and eventually regular page views
 * Citation expansion service API for VE: expand a URL to a full citation
 * HTML load / save API: generalized version of the private VisualEditor API and based on the Parsoid REST API, to be used by VE, Kiwix, external editors, bots, gadgets (see current Parsoid users despite missing save API)
 * Maintain PDF render service

API design and prototyping support

 * Flow REST API for use by rich front-end, mobile; limited prototyping support

Rashomon revision storage service

 * improves VisualEditor performance on both load & save and saves bandwidth for Mobile by replacing Varnish with static storage and removing metadata from the HTML
 * ability to use this for regular page views by spring / summer 2015
 * improved page view performance for editors
 * significantly reduced load on PHP cluster (HW cost and energy savings)
 * seamless and fast switching from page view to VE (edit metadata loaded on demand, HTML used directly)
 * support for cross-datacenter replication, compression and even load distribution across storage cluster

Division of labor between PHP and REST API
The PHP and REST APIs have a different focus and will be largely complementary. The PHP API offers powerful features including generators, but is not designed to support serving content at high request rates. Per-request overheads are high, and caching the output is not generally possible. Mostly due to its performance characteristics it is not commonly consumed by MediaWiki itself.

The REST API on the other hand focuses on simple but high-volume interfaces with a small per-request overhead. It further supports caching, which makes it suitable for directly serving content at a larger scale.

Developers will need clear guidance on which interface they should use for specific tasks. The REST interface will start out fairly small and won't overlap with the PHP API. Over time we can consider gradually migrating simple and high-volume end points (like opensearch) to the REST interface. This will happen in close cooperation with the PHP interface team.

Also see these meeting notes from the API roadmap meeting last September.

Could this be structured as a 'virtual team' like the release management / QA team?
There is a component of helping other teams with API design and early implementations, but the bulk of the work is in the development of services and APIs to be used by other teams.

One important design goal is to achieve sufficient generality to make these APIs useful for several present and future applications. This is easier if the users of these APIs are customers of the API team. Given the expected work load there will also be very little time left to also work on a 'regular team'.

Overall it seems to make more sense to organize this as a real team.