FileBackend design considerations

This project page is for ideas about the FileBackend class rewrite.

The FileBackend project is a refactoring project which will split off backend operations from the FileRepo class hierarchy. The idea is to separate metadata storage from file storage, so that metadata features such as Commons (Foreign*Repo) can be implemented independently from file storage features (Swift etc.), allowing the two to be combined in arbitrary ways.

Applications brainstorming

 * Filesystem
 * Swift
 * Azure
 * Amazon S3

Backend considerations
Current architecture assumes:
 * We can use FS path-like names (called storage paths).
 * If a backend can't do this, it can always normalize them in some fashion.
 * We can list objects/files that have a path starting with a prefix (thumbnail purging uses this)
 * Swift can list object names by prefix
 * php-cloudfiles has a function for these. Also see "swift.common.client.get_container" at http://swift.openstack.org/misc.html#client.
 * Amazon S3 can also list object names by prefix.
 * "Keys can be listed by prefix. By choosing a common prefix for the names of related keys and marking these keys with a special character that delimits hierarchy, you can use the list operation to select and browse keys hierarchically. This is similar to how files are stored in directories within a file system. " -- http://docs.amazonwebservices.com/AmazonS3/latest/dev/
 * Azure supports this too.
 * See "List Blobs" API request at http://msdn.microsoft.com/en-us/library/dd135734.aspx
 * If a backend can't do this, then it will need DB tracking to get the lists.
 * Container support
 * Swift & Azure puts objects in user namable "containers"
 * Amazon S3 puts objects in user "buckets" (basically containers)

Locking considerations
There is a LockManager heirarchy for handling locks.
 * Includes DBFileLockManager class
 * We can use DB-based locking or make a subclass to use some distrubuted lock manger that's already out there. Things like FSRepo or backend uses without involving `image` table rows really need this, since there is no locking at all currently. LocalRepo tries to do some locking by locking `image` table rows.

Performance considerations
On backends without native file/object renaming support, copy+delete has to be used. This kind of sucks for super large files.

FileRepo integration

 * FSRepo and LocalRepo assume an FS backend. We need to change that to remove that assumption but also maintain backwards compatibility, which is a bit tricky.

Other integration

 * thumb.php
 * image_auth.php
 * thumb_handler.php