FileBackend design considerations

This project page is for ideas about the FileBackend class rewrite.

The FileBackend project is a refactoring project which will split off backend operations from the FileRepo class hierarchy. The idea is to separate metadata storage from file storage, so that metadata features such as Commons (Foreign*Repo) can be implemented independently from file storage features (Swift etc.), allowing the two to be combined in arbitrary ways.

Backend considerations
Possible backends:
 * Filesystem
 * Swift
 * Azure
 * Amazon S3

Current architecture assumes:
 * We can use FS path-like names (called storage paths).
 * If a backend can't do this, it can always normalize them in some fashion.
 * We can list objects/files that have a path starting with a prefix (thumbnail purging uses this)
 * Swift can list object names by prefix
 * php-cloudfiles has a function for these. Also see "swift.common.client.get_container" at http://swift.openstack.org/misc.html#client.
 * Amazon S3 can also list object names by prefix.
 * "Keys can be listed by prefix. By choosing a common prefix for the names of related keys and marking these keys with a special character that delimits hierarchy, you can use the list operation to select and browse keys hierarchically. This is similar to how files are stored in directories within a file system. " -- http://docs.amazonwebservices.com/AmazonS3/latest/dev/
 * Azure supports this too.
 * See "List Blobs" API request at http://msdn.microsoft.com/en-us/library/dd135734.aspx
 * If a backend can't do this, then it will need DB tracking to get the lists.
 * Container support
 * Swift & Azure puts objects in user namable "containers"
 * Amazon S3 puts objects in user "buckets" (basically containers)

Locking considerations
There is a LockManager heirarchy for handling locks.
 * Includes DBFileLockManager class
 * We can use DB-based locking or make a subclass to use some distrubuted lock manger that's already out there. Things like FSRepo or backend uses without involving `image` table rows really need this, since there is no locking at all currently. LocalRepo tries to do some locking by locking `image` table rows.

Performance considerations
On backends without native file/object renaming support, copy+delete has to be used. This kind of sucks for super large files.

FileRepo integration

 * FSRepo and LocalRepo assume an FS backend. We need to change that to remove that assumption but also maintain backwards compatibility, which is a bit tricky.
 * Generic zone configuration. The old code has ad-hoc configuration for each zone (directory, deletedDir, thumbDir). It would simplify the FileBackend interface if the relevant code was factored out and the configuration became:
 * array( 'zones' => array( 'public' => array( 'directory' => ..., 'container' => ... ) ) )
 * The old configuration keys would be kept for backwards compatibility.
 * The default backend (FileRepo::getBackend) should be FSFileBackend. Any shared backend interface code should be moved to FileRepo. FSRepo should become an alias for FileRepo.
 * FileRepo can be non-abstract, since the abstract backend access methods will be in FileBackend instead.
 * File::getPath callers should be migrated to some more remote-friendly interface

Other integration

 * Handler code in /media - done
 * Upload code in /upload - done
 * thumb.php
 * image_auth.php
 * thumb_handler.php