User:Brooke Vibber/Media handler and renderer thoughts

From mediawiki.org

Some thoughts on media handlers

Key problem to solution mappings[edit]

At config time:

  • tricky to add new file types from an extension if they're not in the core mime.types
    • add a registry of file extension -> mime type mapping overrides

At upload time:

  • existing upload-time validation tends to be overzealous about .zip-based formats or otherwise gets confused
    • add an upload-time validation function in the Handler

At transform & parser time:

  • srcset is inefficiently handled in Linker
    • allow handlers to do their own srcset sub-transforms
    • add suitable entries to imageinfo for additional resolutions for InstantCommons
  • need reliable way to attach JS modules for file display progressive enhancement
    • for pre-rendered:
      • have a way to transfer list of modules/styles to load from a MediaTransformOutput to a ParserOutput when adding the rendering
    • for dynamic rendered:
      • hang tiny stub modules in on all page views, that hook the on-wikitext-added event and check if they need to load more
  • Rendering remote files of locally-unknown type
    • add common infrastructure for iframe embedding like TMH has
    • expose iframe embed URL in imageinfo response; client side can use it if no local handler available
  • Render files with backing type X but display type Y
    • mark photo spheres, panoramas etc with additional metadata that gets transferred across remote repos, so correct rendering gets picked
    • same can handle things like rendering a PNG/TIFF source to JPEG thumbnails vs PNG depending on photo vs diagram

At MMV popup activation time:

  • Rendering remote files of locally-unknown type
    • Let MMV handle zoom of remote types it doesn't know locally via the iframe embedding

Background: current state of media bits[edit]

  • mime type is primary identifier for picking a Handler class
  • extension -> mime type and other -> mime type mappings are hidden and very hard to extend
  • upload-time validation of file type & safety is hidden and very hard to extend
  • File class encapsulates for a file from a given source repo:
    • how to get the Handler instance
    • wraps the Handler class's metadata & transform processes
      • caches metadata via DB
      • over InstantCommons, ForeignAPIFile speaks to repo over HTTP... up to once per rendering (cached)
  • Handler classes encapsulate for a given file type:
    • how to extract metadata from the file
    • whether rendering to a thumbnail is possible
    • validating thumbnail parameters from wikitext or other source
    • performing some kind of backend rendering
    • returning a suitable Thumbnail subclass instance with a rendering
  • MediaTransformOutput classes encapsulate for a given file type:
    • returning some HTML to actually show in wiki page renderings
    • returning direct references for flat 2d thumbnail image
  • Linker class encapsulates for all types:
    • translating parameter strings into arrays to pass into the handler
    • asking the Handler for thumbnail renderings at 1x, 1.5x, 2x
      • combining the urls from the 1.5x and 2x renderings into the 1x MediaTransformOutput instance
      • note this is hella inefficient over InstantCommons -- 3 separate imageinfo requests get run
    • shoving the rendering into a pretty frame
  • MMV extension encapsulates for some types:
    • how to copy the thumbnail into a big pretty viewer, then ask for a high-res thumbnail and switch to it when loaded
    • is based on file extension of the thumbnail, not on mime type
  • ad-hoc JavaScript modules attached to particular types may handle:
    • progressive enhancement of the displayed thumbnail
      • TMH's replacement of bare <video>/<audio> or 'popup thumbnail' with fancy player widget
    • progressive enhancement of the MMV high-res view
      • 3d STL extension's in-progress plugin
      • currently MMV displayable-ness and plugin mapping are based on file extension of the thumbnail, which seems fragile
    • currently TMH jumps through some hoops to mark ParserOutput for later loading of the modules
    • no clear way to handle adding the modules dynamically when dynamically adding source through a runtime preview, etc

Ok that was kinda scary, huh?