User:Brion VIBBER/Media handler and renderer thoughts

Some thoughts on media handlers

Key problem to solution mappings
At config time:
 * tricky to add new file types from an extension if they're not in the core mime.types
 * add a registry of file extension -> mime type mapping overrides

At upload time:
 * existing upload-time validation tends to be overzealous about .zip-based formats or otherwise gets confused
 * add an upload-time validation function in the Handler

At transform & parser time:
 * srcset is inefficiently handled in Linker
 * allow handlers to do their own srcset sub-transforms
 * add suitable entries to imageinfo for additional resolutions for InstantCommons
 * need reliable way to attach JS modules for file display progressive enhancement
 * for pre-rendered:
 * have a way to transfer list of modules/styles to load from a MediaTransformOutput to a ParserOutput when adding the rendering
 * for dynamic rendered:
 * hang tiny stub modules in on all page views, that hook the on-wikitext-added event and check if they need to load more
 * Rendering remote files of locally-unknown type
 * add common infrastructure for iframe embedding like TMH has
 * expose iframe embed URL in imageinfo response; client side can use it if no local handler available
 * Render files with backing type X but display type Y
 * mark photo spheres, panoramas etc with additional metadata that gets transferred across remote repos, so correct rendering gets picked
 * same can handle things like rendering a PNG/TIFF source to JPEG thumbnails vs PNG depending on photo vs diagram

At MMV popup activation time:
 * Rendering remote files of locally-unknown type
 * Let MMV handle zoom of remote types it doesn't know locally via the iframe embedding

Background: current state of media bits
Ok that was kinda scary, huh?
 * mime type is primary identifier for picking a Handler class
 * extension -> mime type and other -> mime type mappings are hidden and very hard to extend
 * upload-time validation of file type & safety is hidden and very hard to extend
 * File class encapsulates for a file from a given source repo:
 * how to get the Handler instance
 * wraps the Handler class's metadata & transform processes
 * caches metadata via DB
 * over InstantCommons, ForeignAPIFile speaks to repo over HTTP... up to once per rendering (cached)
 * Handler classes encapsulate for a given file type:
 * how to extract metadata from the file
 * whether rendering to a thumbnail is possible
 * validating thumbnail parameters from wikitext or other source
 * performing some kind of backend rendering
 * returning a suitable Thumbnail subclass instance with a rendering
 * MediaTransformOutput classes encapsulate for a given file type:
 * returning some HTML to actually show in wiki page renderings
 * returning direct references for flat 2d thumbnail image
 * Linker class encapsulates for all types:
 * translating parameter strings into arrays to pass into the handler
 * asking the Handler for thumbnail renderings at 1x, 1.5x, 2x
 * combining the urls from the 1.5x and 2x renderings into the 1x MediaTransformOutput instance
 * note this is hella inefficient over InstantCommons -- 3 separate imageinfo requests get run
 * shoving the rendering into a pretty frame
 * MMV extension encapsulates for some types:
 * how to copy the thumbnail into a big pretty viewer, then ask for a high-res thumbnail and switch to it when loaded
 * is based on file extension of the thumbnail, not on mime type
 * ad-hoc JavaScript modules attached to particular types may handle:
 * progressive enhancement of the displayed thumbnail
 * TMH's replacement of bare / or 'popup thumbnail' with fancy player widget
 * progressive enhancement of the MMV high-res view
 * 3d STL extension's in-progress plugin
 * currently MMV displayable-ness and plugin mapping are based on file extension of the thumbnail, which seems fragile
 * currently TMH jumps through some hoops to mark ParserOutput for later loading of the modules
 * no clear way to handle adding the modules dynamically when dynamically adding source through a runtime preview, etc