Requests for comment/PHP Virtual REST Service

Problem description
Our current PHP bindings for the use of REST-style (web) services are suffering from several issues:


 * Per-service issues like authentication need to be dealt with manually for each request.
 * Multiple requests can't easily be performed in parallel, which prevents us from leveraging parallelism to speed up request processing.
 * There is no transparent and generic load balancing and fail-over support. This can also be worked around with extra infrastructure (separate load balancers, automatic failover setups), but this comes at the price of extra latency and complexity.
 * Transparently changing service implementations to use a local class, other network protocols or a mock server is not well supported. Combined with the common use of global variables for configuration variables like host names this makes unit testing very difficult. Backend optimizations often require changes to all code using the service.

With more web services in the backend infrastructure it might be worth spending some time on raising the level of abstraction in our interaction with REST services from PHP.

Goals

 * Convenient to use : Handle repetitive low-level detail in backend handlers.
 * Don't obfuscate REST interfaces : Expose REST style interfaces faithfully so that developers don't need to learn several names for the same thing.
 * Provide a Virtual REST Service that abstracts over protocols and backends : Use a high-level interface that captures the interaction with the service interfaces, but don't hard-code how those services are accessed or implemented (compare: VFS).
 * Be open to the world : Also support use as a simple CURL client, but encourage the use of more convenient and abstracted backends for ease of use, performance and testability.

API sketch
This is a very early strawman API design that maps fairly easily onto a curl_multi based implementation:

Backend handlers
The generic store has a registry of prefixes to storage backend handlers. Backends can be HTTP-based, but can also be transparently mapped to local PHP classes or other protocols. In the case of backends that map to protocols curl supports, the handler needs to 1) convert a simple path request to a full curl request, and 2) handle responses coming back from curl. Non-curl protocols will additionally require an integration with the curl_multi loop, which is fairly straightforward if they provide a non-blocking 'do some work' interface similar to curl_multi.

Matching happens longest-prefix first. Very rough example (implementation details to be worked out):

Using paths for dispatching lets us set up a simple and unified namespace. Backend implementations can be reused for several services by instantiating them for each service with a different prefix.

The thin wrapper over REST-style interfaces makes new features in those directly available without a need to upgrade client bindings. Bulk operations can be performed across several backends.

Tasks to be handled by backend implementations:

Outgoing request setup

 * authentication: retrieve / refresh auth key, add auth headers etc
 * URL and more generally request munging
 * Select form encoding for post requests ( vs  )
 * Accept-encoding default if not specified in request
 * Add Content-MD5 for PUT requests

It should be possible to implement this by passing the full request information to each backend for general munging.

It might be worth considering future support for asynchronous auth key retrieval, even if the initial implementation can be kept simple by doing this synchronously. The key will be cached in the backend object after the first request and will likely not need refresh during a single PHP request.

Response handling

 * error handling: auth refresh, retry, reporting
 * GZIP transfer-encoding handling if not done by HTTP library backend used
 * MWHttpRequest does not yet support it, but post-2011 curl can with the right option
 * JSON parsing for  (with request option to disable)