Multi-Content Revisions/Blob Storage

From MediaWiki.org
Jump to navigation Jump to search
This page is part of the MCR proposal. Status: stable draft, comments welcome. ExternalStore Integration and Structured Storage are still in early draft stage.

The lowest level of the storage infrastructure is the blob storage service, which allows arbitrary binary data to be stored and retrieved. We want to be able to have several different such storage facilities available, and address them by name. In particular, it should be possible to use different storage backends for different slots, or different content models.

For each storage mechanism, a BlobStore implementation would be provided:

interface BlobLookup {
	 /**
	 * @param string $address The desired blob's address, as returned by BlobStore::storeData().
	 * @param int $queryFlags Bitfield, see the IDBAccessObject::READ_XXX constants.
	 *
	 * @return string the binary data
	 */
	public function loadData( $address, $queryFlags = 0 );
}

interface BlobStore extends BlobLookup {
	/**
	 * @param string $data binary data to store
	 * @param array $hints associative array of hint keys to hint values.
	 *
	 * @todo Add a transaction context as a parameter.
	 *
	 * @return string the permanent canonical address of the blob. Can be used
	 *         with BlobLookup::loadData().
	 */
	public function storeData( $data, $hints = [] );
}

(Code experiment: https://gerrit.wikimedia.org/r/#/c/299020)

  • The BlobStore has full control over the address that shall be used later to retrieve the blob. storeData() returns the address that can be used with loadData() to retrieve the blob.
  • The address is completely opaque. It may be based on the blob's content hash, use incremental numbering, or GUIDs, or some other scheme.
  • Hints are used to expose meta-data to the storage layer to allow for optimization (e.g. revisions from the same page can be grouped together for compression).

Managing multiple storage backends[edit]

The mapping between slots and storage backends is implemented in two steps:

  • BlobStore names are associated with BlobStore implementations and configurations, designating a concrete storage location. This association must NEVER change, otherwise any stored data will become inaccessible (this is similar to how externalstore clusters are configured). This mapping is managed by BlobStoreMux (see below).
  • slot names are associated with a BlobStore name. This indicates which store is to be used when storing new data. This association can be changed at will. This mapping is not used by the blob storage layer, it is only needed when saving content for a new revision.

The string returned by storeData() is an opaque URL for later loading the slot data using loadData().

/**
* Stores data to one of several underlying BlobStores, based on a logical name.
* Provides blob address resolution based on the logical name of a blob store
* encoded in a blob address.
*
* @todo find a better name. BlobAddressResolver? BlobStorageManager?
* @todo should this be a BlobStoreRegistry, or have a BlobStoreRegistry?
**/
interface BlobStoreMux extends BlobLookup {

	/**
	 * @param string $storeName the logical name of the BlobStore to save the data to.
	 * @param string $data binary data to store
	 * @param array $hints associative array of hint keys to hint values.
	 *
	 * @todo Add a transaction context as a parameter.
	 *
	 * @return string the permanent canonical address of the blob. Can be used
	 *         with BlobStoreMux::loadData(). The address will typically contain
	 *         the store name as a prefix, and the address returned by the
	 *         underlying store as a suffix.
	 */
	public function storeDataTo( $storeName, $data, $hints = [] );
}

(Related code experiment: https://gerrit.wikimedia.org/r/#/c/217710/)

BlobStoreMux manages a number of BlobStores by logical name, and provides address resolution based on such names. Specifically:

  • The address returned by storeDataTo is composed of two parts: the name of the BlobStore, and the address returned by the BlobStore.
  • BlobStoreMux' implemention of loadData() relies on the prefix in the $address to find the correct BlobStore to load the blob.

Note: storeData() and storeDataTo() should probably get some kind of transaction context as an additional parameter. See Transaction Management.

Initial BlobStore Implementation[edit]

The initial BlobStore implementation will be based on the text table. It will use the relevant code from the getRevisionText() and insertOn() in the Revision class, including support for compression, transcoding, and External Storage. Later, support for ExternalStorage can be improved by bypassing the text table alltogether, and using the external storage URL as the blob address. See the section on intergation with ExternalStore below.

Besides the SQL-Based storage used currently used by MediaWiki, BlobStores could be implemented on top of the raw file system, Cassandra, Apache Swift, higher level HTTP based services like RESTBase, etc.

Hints[edit]

Storage hints provide a way to "leak" high level information to the blob storage layer to allow for optimization. Similarly, hints can be used by the blob storage layer to expose meta-data to higher level code. All hints are optional, all storage operations should function without them.

Some hints that may be useful:

  • model: Hint at the content model of the data. BlobStores may use this to optimize storage for well known content models.
  • format: Hint at the serialization format of the data. BlobStores may use this to optimize storage for well known data formats.
  • hash: Hint at the hash of the data. BlobStores that need a content hash may use this instead of re-calculating the hash.
  • page: Hint at the page the data belongs to. BlobStores may use this to group related data together, e.g. for prefetching.
  • revision: Hint at the revision the data belongs to. BlobStores may use this to group related data together, e.g. for prefetching.
  • replace: Hint at an address of a blob that is superseded by the new data. BlobStores may use this to discard obsolete data.
  • similar: Hint at an address of a blob that is probably similar. BlobStores may use this to group similar data together, e.g. for compression.
  • parent: Hint at the parent revision of the revision the data belongs to. BlobStores may use this to group similar data together, e.g. for compression.
  • used-by: Hint at the URI of a resource that uses the blob. BlobStores may use this for reference counting.

Integration with ExternalStore[edit]

The introduction of BlobStore is not blocked on integrating with ExternalStore, nor is the introduction of Content Meta-Data. An initial BlobStore implementation can use ExternalStore as-is, including the indirection via the text table.

However, to avoid the indirection of storing ExternalStore URLs in the text table, ExternalStore should be integrated with the new BlobStore infrastructure. This poses some minor challanges: The class ExternalStoreMedium exposes similar methods as the BlobStoreMux interface descibed above:

 public function fetchFromURL( $url );
 public function store( $location, $data );

However, ExternalStoreMedium is not accessed directly; instead, the ExternalStore class is used to do the multiplexing between different ExternalStoreMedium objects. It exposes the following relevant methods:

 public static function fetchFromURL( $url, array $params = [] )
 public function insertWithFallback( array $tryStores, $data, array $params = [] )

Note that insertWithFallback doesn't get one store, but a fallback chain of stores; this doesn't seem to be used in practice, though.

The $params argument means that for each call to insertWithFallback or fetchFromURL, a new ExternalStoreMedium instance is created based on these parameters. The main purpose of $params seems to be to select a wiki, in case we are trying to load blobs that belong to another site. With the BlobStoreMux interface, each service instance would be permanent, and bound to a specific wiki. To access blobs for a different wiki, an appropriate BlobStoreMux instance would have to be acquired from a factory.

When introducing BlobStore, the existing older interface needs to be integrated. There seem to be three options:

  1. Don't introduce a new blob storage interface at all, expand and adopt the existing ExternalStore facility.This would be a considerable logical break: ExternalStore would no longer be an implementation detail hidden by the code that manages the text table - to the contrary, it would be the primary way to access content data, and the text table would just be one possible ExternalStoreMedium. Addresses for direct storage in the text table will look something like TT:7641432, externally stored blobs would keep using addresses like DB://cluster5/873284.
  2. Turn ExternalStore into a BlobStoreMux, in addition to the purely static interface. This means two-level multiplexing, once for the BlobStore interface, and once for the ExternalStoreMedium interface. Blob addresses using the external store would look something like ES:DB://cluster5/873284. (Code experiment: https://gerrit.wikimedia.org/r/#/c/300533)
  3. Create an ExternalStoreMediumBlobStore adapter, that implements the BlobStore interface on top of an ExternalStoreMedium instance. The BlobStoreMux would use these adapters directly, bypassing the old EntityStore class completely. Blob addresses from this adapter would be the same as the old external store URLs, e.g. DB://cluster5/873284.

Note that with the content meta-data storage in place, we no longer need to write the URL to the text table, since it can be stored in the cont_address field directly. However, when converting from ES-URL-in-the-text-table to ES-URL-in-cont_address, care must be taken to encode the information in old_flags into the URL, e.g. DB://cluster5/873284;latin1,gzip.

Structured Storage[edit]

In the future, it may become desirable to store content objects in a structured way, instead of blobs. For example, tabular data could be stored in a relational database table, workflow state could be stored in a document oriented database, etc. To allow this, we would need a storage level interface that handles Content objects instead of blobs:

interface BlobLookup {
	 /**
	 * @param string $address The desired content's address, as returned by ContentStore::storeData().
	 * @param int $queryFlags Bitfield, see the IDBAccessObject::READ_XXX constants.
	 *
	 * @return Content 
	 */
	public function loadContent( $address, $queryFlags = 0 );
}

interface ContentStore extends ContentLookup {
	/**
	 * @param Content $content content to store
	 * @param array $hints associative array of hint keys to hint values.
	 *
	 * @todo Add a transaction context as a parameter.
	 *
	 * @return string the permanent canonical address of the content. Can be used
	 *         with ContentLookup::loadContent().
	 */
	public function storeContent( $content, $hints = [] );
}

And similarly, a ContentStoreMux class, analogous to BlobStoreMux.

Higher level code could either always go through the content based interface, falling back to a generic implementation of ContentStore based on serialized storage.

TBD: it may be more useful to place the structured storage interface at a slightly higher level of abstraction, so it doesn't use opaque addresses, but has access to revision ID and slot name: function loadContent( $revision, $slot, $queryFlags = 0 ). This would allow for virtual slots to be implemented naturally using this interface, though it would not solve the problem of a virtual slot implementation needing access to the primary slot content (and perhaps also content of the parent revision, for blame maps and diffs).

(TBD: This perhaps fits better with Multi-Content Revisions/Revision Retrieval than here)

Batch Interface[edit]

A batch interface will likely be required in the future, at least for loading, but perhaps also for writing blobs (e.g. when importing a dump). The batch methods should probably be defined in separate interfaces:

  • BatchContentLookup::loadDataBatch
  • BatchContentStore::storeDataBatch
  • BatchContentStoreMux::storeDataBatchTo