Core Platform Team/Initiatives/Mainstash Multi-DC

Product requirements

 * Persistent data storage (not LRU or other space-bounded caching)
 * General-case storage
 * Specific requirements for particular components will be handled separately (e.g. session storage)
 * Most components cannot be configured to use different storage
 * Key-value storage matching BagOStuff interface
 * key type: string
 * value type: string, integer, object, array
 * Client-set time-to-live (TTL)
 * Atomic increment
 * Set-if-not-exists (setnx, add)
 * Check-and-set
 * lock/unlock keys
 * get/set/delete batch
 * Multi-data-centre access
 * read-write in primary data centre
 * read-only in secondary data centre(s)
 * a single Web request uses only one data centre
 * access to different data centres may occur in the same Web session
 * All requests for write HTTP methods ( (POST/PUT/DELETE) go to primary data centre
 * "datacenter_preferred" cookie provides ~10s affinity for primary data centre after write request
 * Synchronous access for web requests
 * 10^0 OOM reads per request
 * 10^0 OOM writes per request
 * Performance
 * Current single-DC solution is Redis (10^0 OOM get, 10^2 OOM set)