Reading/Multimedia/Architecture/Tech Debt Backlog

Loosely prioritized list of technical debt projects related to the Multimedia team.

Backlog
FIXME

2013-08-23 Grooming Meeting

 * User:Bawolff, User:BDavis_(WMF), User:MarkTraceur, User:Fabrice_Florin_(WMF)
 * Discussed high level issues
 * Helped Fabrice understand terms "Varnish" and "Squid" and caching in general
 * Discussed difference between a feature and tech debt
 * User:Bawolff gave his top item list including relative priority:
 * Chunked upload. Session and general more reliability
 * Upload class (more extensible. The chunked uploading case is very ugly)
 * deadlocks and long held locks in filebackend / file class when moving/deleting/uploading file.
 * current image thumbnail should have timestamp in it for better caching clear
 * Partial failures in filebackend (If things fail, either nothing should happen or everything should happen. Sometimes pages get deleted but not the file, or vice veras)
 * Various swift layer improvements/thumbnailing cache (making thumbs cached at a different layer, maybe. Making content at an sha1 place and pretty urls only to user)

Copied from Multimedia/Architecture
Copied list and discussion from prior home on Multimedia/Architecture.


 * API upload is all kind of bad. everything under the directory /upload would be nice to rewrite.
 * though please be sure to test well with Upload Wizard and other API consumers, it's very easy to accidentally break edge cases (e.g. blacklist failures, duplicate checks, yadda yadda yadda) -- User:Eloquence
 * backlink invalidation in commons would be nice to fix (22390)
 * Redirect of old filename on file move would be nice. (for hotlinkers like the wmf blog, and due to lack of purging cache of commons client wikis)
 * this is hard to implement as a general MediaWiki feature without changing to run files through PHP by default, but could be done with our special file backends... -- User:Brion_VIBBER
 * Our use case is probably the most important. I don't think third parties use file redirects that much. Patch https://gerrit.wikimedia.org/r/#/c/80135/ -- User:Bawolff
 * versioned urls for thumbnails
 * large file support in general - deadlocks abound
 * way FileRepo stores files is bad in general...requires pessimistic in general (see Requests for comment/Refactor on File-FileRepo-MediaHandler)
 * Why chunked upload should be fixed:
 * Upload actions that go through the job queue should be more reliable.
 * Suggest that long term stop using session data for this (people log in, log out, etc).
 * possible several pieces here: fix job queue to be sane, fix upload jobs to not (ab?)use session storage, and ? -- User:Brion_VIBBER
 * HTCP purges need active monitoring - DONE

Resources
People to talk to about things and stuff:
 * Aaron -- all things cache related
 * User:Bawolff -- all the MM things
 * Brad -- API
 * Faidon -- file storage infrastructure, imagescalers, operational/systems aspects in general