Wikimedia Engineering/Service and REST API team

One of outcomes of the architecture summit in January was a consensus that we should move towards a service-oriented architecture. Major reasons for this are:


 * enabling new engineers to be more productive by letting them concentrate on a part of the stack
 * improved reliability and security through fault and security context isolation
 * ease of testing, debugging, monitoring and scaling
 * supporting new uses of our data by rich clients and mobile through a well-structured and efficient public content API

A storage service and a REST content API were seen as particularly good candidates for next steps in this direction. There has been some preliminary work on the Rashomon revision storage service by the Parsoid team and a generic PHP service interface with support for parallel operations by Aaron.

This has happened as side projects, with other team responsibilities taking priority. We need to do more if we really want to reap the benefits of a service-oriented architecture.

The current budget for next fiscal provides for a small team (4 engineers) to move this forward. This team will


 * guide existing work to think in terms of independent services and versioned APIs,
 * help to develop internal and external REST APIs (ex: Flow),
 * help to set up automated tests against those APIs,
 * identify and implement general backend services in support of other teams (ex: storage service), and
 * support packaging for internal and third party distribution.

In the short term, this can help other teams by removing some of their workload, avoiding duplicate work and helping with API implementations.

In the longer term, it will
 * build up a consistent REST content API to enable mobile and innovative feature projects,
 * provide back-end services like the Rashomon storage service, and
 * improve testing and thus the ability to deploy more often without breaking things.

Ultimately, it should help us make real progress towards a service-oriented architecture with strong APIs. It will hopefully also result in improved developer productivity, testability, security and agility as discussed in the SOA RFC.

= Plans for next fiscal =

Public REST API

 * Frontend (restface)
 * HTML load / save API: generalized version of the private VisualEditor API and based on the Parsoid REST API, to be used by VE, Kiwix, external editors, bots, gadgets (see current Parsoid users despite missing save API)
 * Page metadata API for rendering of red links and other bits in Flow, VE, Mobile and eventually regular page views
 * Citation expansion service API for VE: expand a URL to a full citation
 * CentralNotice banner service
 * Caching & reliable purging for public API

API design and prototyping support for other teams

 * Help the Flow team in the development of a REST API for use by rich front-end, mobile
 * possibly help to refactor Flow backend to take more advantage of existing infrastructure, improve performance

Backend services

 * Storage service (see below)
 * Maintain PDF render service
 * Maintain Math render service (Mathoid)
 * Queuing service (job runner)

Storage service

 * important for performance: VE async save, static HTML for mobile front-end & authenticated users
 * ability to use this for regular page views autumn 2014
 * improved page view performance for editors
 * significantly reduced load on PHP cluster (HW cost and energy savings)
 * seamless and fast switching from page view to VE, async saving
 * support for cross-datacenter replication, compression and even load distribution across storage cluster

Generalization of storage service to support different bucket types

 * Candidate bucket types, roughly by priority: versioned blob, queue, key-value, ordered key-value, counter
 * Features like authentication, TTL

Update & invalidation jobs

 * Ensure that stored data is kept up to date with changes, and front-end caches are invalidated
 * Do we write a service layer around the existing job queue?
 * If so, we should have a new job type that is just a method to call a REST URL

Drive automated service testing

 * Goal: New public APIs should be stable, versioned, and fully covered
 * The frontend should be integration tested with it's backend -- either mocked or in beta labs


 * Working with QA & Antoine to set up automated testing w/ specific node versions
 * Containerization
 * What test framework are we going to use? npm test?
 * Wrapping PHP services? -- Do we treat them differently
 * Integration testing vs unit testing -- we will need to test the frontend wrapper as well as the backend implementation

Structured API documentation

 * Goal: Enable third party & internal app, gadget and bot development
 * Help establish best practices in declarative API documentation using tools like swagger

Service authentication in collaboration with platform
Work with platform on an authentication solution that will work well in a SOA world.

Deployment and Packaging in collab with platform, ops

 * Drive packaging of services for practical third-party and internal use
 * Leverage packages as much as possible for deployment, DRY
 * Provide solid configuration management with Puppet
 * Develop best practices for packaging

Stretch goals

 * Provide a simplified job queue using queue buckets in the storage service layer
 * REST request runner for tasks like HTML pre-generation

caching and purging

 * Do we have varnish redirect?
 * Can we have the API stream a result after it looks up the latest version

Q4 / Q1

 * REST API front-end (restface): (G: 1-2m)
 * PDF: (M: 2m)
 * Rashomon (with versioned blob and queue buckets & lame auth) (G: 1-1.5m)
 * Citation service (M,G: 2d-3w)
 * Mathoid deploy (M: 2w, G: 1w)
 * Wrap HTML load/save in restface (M: 2w, G: 1w)
 * Templating: documentation!!!! (G: 1d)

Q2

 * More intelligent backend authentication (G: 2m)
 * Page metadata for Parsoid HTML views (redlinks etc) (M: 1m)
 * Caching and purging (w/ help from ops) (G: 2w)
 * Packaging / deployment -- Make debian/ubuntu packages for frontend / pdf / rashomon (M: 2w)
 * Documentation -- flesh out the registered structured handler bit, create frontend (G: 2w)

Q3

 * Job queue runner (M,G: 2w)
 * Help with mobile API / service needs (M: ???)
 * Flow API needs (M: ???)
 * New bucket types in StoreKeeper

Q4

 * CentralNotice (M: 1m)
 * Echo?
 * Multi datacentre operation
 * cacheable WikiData API?

= FAQ =

Division of labor between PHP and REST API
The PHP and REST APIs have a different focus and will be largely complementary. The PHP API offers powerful features including generators, but is not designed to support serving content at high request rates. Per-request overheads are high, and caching the output is not generally possible. Mostly due to its performance characteristics it is not commonly consumed by MediaWiki itself.

The REST API on the other hand focuses on simple but high-volume interfaces with a small per-request overhead. It further supports caching, which makes it suitable for directly serving content at a larger scale.

Developers will need clear guidance on which interface they should use for specific tasks. The REST interface will start out fairly small and won't overlap with the PHP API. Over time we can consider gradually migrating simple and high-volume end points (like opensearch) to the REST interface. This will happen in close cooperation with the PHP interface team.

Also see these meeting notes from the API roadmap meeting last September.