Multi-Content Revisions/Blob Storage

The lowest level of the storage infrastructure is the blob storage service, which allows arbitrary binary data to be stored and retrieved. We want to be able to have several different such storage facilities available, and address them by name. In particular, it should be possible to use different storage backends for different slots, or different content models.

The general interface for storing data blobs is defined by the BlobStore interface:


 * The BlobStore has full control over the address that shall be used later to retrieve the blob. storeBlob returns the address that can be used with getBlob to retrieve the blob.
 * The address is completely opaque. It may be based on the blob's content hash, use incremental numbering, or GUIDs, or some other scheme.
 * Hints are used to expose meta-data to the storage layer to allow for optimization (e.g. revisions from the same page can be grouped together for compression).

Managing multiple storage backends
The mapping between slots and storage backends can be split into two steps: The string returned by storeBlob is an opaque URL for later loading the slot data using getBlob.
 * BlobStore names are associated with BlobStore implementations and configurations, designating a concrete storage location. This association must NEVER change, otherwise any stored data will become inaccessible (this is similar to how externalstore clusters are configured). This mapping is managed by BlobStoreMux (see below).
 * slot names are associated with a BlobStore name. This indicates which store is to be used when storing new data. This association can be changed at will. This mapping is not used by the blob storage layer, it is only needed when saving content for a new revision.

(Related code experiment: https://gerrit.wikimedia.org/r/#/c/217710/)

BlobStoreMux manages a number of BlobStores by logical name, and provides address resolution based on such names. Specifically: Note: storeBlob and storeBlobTo should probably get some kind of transaction context as an additional parameter. See Transaction Management.
 * The address returned by storeBlobTo is composed of two parts: the name of the BlobStore, and the address returned by the BlobStore.
 * BlobStoreMux' implemention of getBlob relies on the prefix in the $ to find the correct BlobStore to load the blob.

Basic BlobStore Implementation
The basic BlobStore implementation is based on the  table. It has support for compression, transcoding, and External Storage. Later, support for ExternalStorage can be improved by bypassing the text table alltogether, and using the external storage URL as the blob address. See the section on intergation with ExternalStore below.

Besides the SQL-Based storage used currently used by MediaWiki, BlobStores could be implemented on top of the raw file system, Cassandra, Apache Swift, higher level HTTP based services like RESTBase, etc.

Hints
Storage hints provide a way to "leak" high level information to the blob storage layer to allow for optimization. Similarly, hints can be used by the blob storage layer to expose meta-data to higher level code. All hints are optional, all storage operations should function without them.

Some hints that may be useful:
 * model: Hint at the content model of the data. BlobStores may use this to optimize storage for well known content models.
 * format: Hint at the serialization format of the data. BlobStores may use this to optimize storage for well known data formats.
 * hash: Hint at the hash of the data. BlobStores that need a content hash may use this instead of re-calculating the hash.
 * page: Hint at the page the data belongs to. BlobStores may use this to group related data together, e.g. for prefetching.
 * revision: Hint at the revision the data belongs to. BlobStores may use this to group related data together, e.g. for prefetching.
 * replace: Hint at an address of a blob that is superseded by the new data. BlobStores may use this to discard obsolete data.
 * similar: Hint at an address of a blob that is probably similar. BlobStores may use this to group similar data together, e.g. for compression.
 * parent: Hint at the parent revision of the revision the data belongs to. BlobStores may use this to group similar data together, e.g. for compression.
 * used-by: Hint at the URI of a resource that uses the blob. BlobStores may use this for reference counting.

Integration with ExternalStore
The introduction of BlobStore is not blocked on integrating with ExternalStore, nor is the introduction of Content Meta-Data. An initial BlobStore implementation can use ExternalStore as-is, including the indirection via the text table.

However, to avoid the indirection of storing ExternalStore URLs in the text table, ExternalStore should be integrated with the new BlobStore infrastructure. This poses some minor challanges: The class ExternalStoreMedium exposes similar methods as the BlobStoreMux interface descibed above:

However, ExternalStoreMedium is not accessed directly; instead, the ExternalStore class is used to do the multiplexing between different ExternalStoreMedium objects. It exposes the following relevant methods:

Note that insertWithFallback doesn't get one store, but a fallback chain of stores; this doesn't seem to be used in practice, though.

The $params argument means that for each call to insertWithFallback or fetchFromURL, a new ExternalStoreMedium instance is created based on these parameters. The main purpose of $params seems to be to select a wiki, in case we are trying to load blobs that belong to another site. With the BlobStoreMux interface, each service instance would be permanent, and bound to a specific wiki. To access blobs for a different wiki, an appropriate BlobStoreMux instance would have to be acquired from a factory.

When introducing BlobStore, the existing older interface needs to be integrated. There seem to be three options:

Note that with the content meta-data storage in place, we no longer need to write the URL to the text table, since it can be stored in the  field directly. However, when converting from ES-URL-in-the-text-table to ES-URL-in-cont_address, care must be taken to encode the information in  into the URL, e.g..
 * 1) Don't introduce a new blob storage interface at all, expand and adopt the existing ExternalStore facility.This would be a considerable logical break: ExternalStore would no longer be an implementation detail hidden by the code that manages the text table - to the contrary, it would be the primary way to access content data, and the text table would just be one possible ExternalStoreMedium. Addresses for direct storage in the text table will look something like , externally stored blobs would keep using addresses like.
 * 2) Turn ExternalStore into a BlobStoreMux, in addition to the purely static interface. This means two-level multiplexing, once for the BlobStore interface, and once for the ExternalStoreMedium interface. Blob addresses using the external store would look something like  . (Code experiment: https://gerrit.wikimedia.org/r/#/c/300533)
 * 3) Create an ExternalStoreMediumBlobStore adapter, that implements the BlobStore interface on top of an ExternalStoreMedium instance. The BlobStoreMux would use these adapters directly, bypassing the old EntityStore class completely. Blob addresses from this adapter would be the same as the old external store URLs, e.g..

Structured Storage
In the future, it may become desirable to store content objects in a structured way, instead of blobs. For example, tabular data could be stored in a relational database table, workflow state could be stored in a document oriented database, etc. To allow this, we would need a storage level interface that handles Content objects instead of blobs:

And similarly, a  class, analogous to.

Higher level code could either always go through the content based interface, falling back to a generic implementation of ContentStore based on serialized storage.

TBD: it may be more useful to place the structured storage interface at a slightly higher level of abstraction, so it doesn't use opaque addresses, but has access to revision ID and slot name:. This would allow for virtual slots to be implemented naturally using this interface, though it would not solve the problem of a virtual slot implementation needing access to the primary slot content (and perhaps also content of the parent revision, for blame maps and diffs).

(TBD: This perhaps fits better with Multi-Content Revisions/Revision Retrieval than here)

Batch Interface
A batch interface will likely be required in the future, at least for loading, but perhaps also for writing blobs (e.g. when importing a dump). The batch methods should probably be defined in separate interfaces: