User:Brion Vibber (WMF)/Mobile video playback 2023

Media playback, and in particular video playback, is broken or partially broken in the mobile apps and can be improved. I'm embedding with the combined mobile team's sync meetings to plan some work to fix this; currently laying out the possible things we can do so we can decide on best trade-off.

Basic issues

 * complete lack of native WebM and Ogg support in and on iOS web views
 * bare and on Android may not pick the best resolution
 * bare links to Ogg or Opus audio source files are common in pronunciation templates, so even though we produce an MP3 transcode it's not used in this circumstance on iOS unless the frontend JS knows to look for it

todo:
 * double-check current state of everything in iOS and Android app releases, see if there's any additional problems with TimedMediaHandler specifically
 * double-check 3d model behavior (isolated in files like video/audio, but different thumbnail output)
 * double-check Graphs behavior (not isolated in files, may require different handling)

Possible remediations
Main possibilities in order of increasing complexity:

iframe embed
Have in-app JS replace the bare players with s pointing to the embedded version in the web UI


 * good: this will load the necessary player UI and compatibility shim code to work on iOS
 * bad: note that decode performance will be single-threaded and likely less battery-efficient than native playback with hardware acceleration
 * good: it's future-proof against future improvements to re-encode files in a format that both iOS and other operating systems/browsers are happy with (planned for later)
 * bad: won't work offline (may be fine?)
 * good: loose coupling, the  param could be reused on other file types
 * bad: however this is not currently implemented. probably deserves first-class support in MediaWiki.
 * bad: the same system may not work as well for things like Graphs that don't have a separate page where they live; this needs further investigation
 * bad: no app customization of the UI inside the iframe, unless we devise an API that communicates over postMessage or query parameters

Load ResourceLoader modules into the app
Load the video.js player UI, including its ogv.js compat shim on iOS, into the web view from ResourceLoader, bundling it for offline caching


 * bad: may require retooling the player setup functions to work in the mobile app environment
 * ?: not sure if we are currently doing the 'pull a module from ResourceLoader and save it' sensibly or not :D
 * bad: sounds like tight coupling, specific to TimedMediaHandler
 * good: could be future-proofed to be more generic, exposing a specific list of modules that are safe to load outside mediawiki and needed for a given page
 * good: app could customize the player a bit via styles or plug-in JS

Native app player
Parse the sources lists from the or and present the selected resolution/codec via native code


 * bad: AVFoundation does not support WebM on iOS so this requires bundling at least WebM and Ogg demuxers and Vorbis and Opus audio codecs
 * VP8, VP9 should be available on modern iOS devices for hardware decoding via VideoToolbox or indirectly through AVSampleBufferDisplayLayer starting in iOS 14 or 15 (IIRC) but older devices don't have hardware decoders at least for VP8. it may still be necessary to ship libvpx, which is free software and royalty-free like libopus and libvorbis
 * good: VLC has done all this work already! It's possible to build a MobileVLCKit library (optionally with non-free codecs excluded), that exposes a sensible widget API, including support for only the codecs you need.
 * bad: however it will be desirable to tune the build to include the minimum of format and codec support required, and we'd need to occasionally update that build from upstream, so this doesn't happen without labor
 * bad: doesn't help for other types of media that need to ship "behaviors" (code) with their HTML, such as Graphs
 * good: complete app control over the player UI

Short term: choices

 * is offline important?
 * if no, iframe is functional and future-proof
 * if yes, need to access files & either bundle JS/wasm decoder or native decoder
 * is Graphs important now?
 * if so, probably need a non-iframe JS module
 * alternately, ability to put code in iframe and inject data from the host document (consider security)

Short term: re-encoding for iOS video
My current work patch is doing some fixup to video transcode output which will include an experimental 144p QuickTime MJPEG/MP3 output. This is a low resolution because compression with motion-JPEG is awful, but since JPEG and MP3 are things we already support, and QuickTime/MP4/ISO BMFF is 'standard enough' and widely implemented, it works within our existing rules and more importantly *works on Safari* on all iOS and Mac devices, even the old ones without newer VP9 hardware codecs available, and even when our JS doesn't run to load the WebAssembly decoder shim.

If all goes well I hope to start a batch job to roll out fallback files for all existing uploads and they'll just 'start working'. Other improvements we make in future will give higher res videos, or better interaction, or both.

Long term: re-encoding for iOS video
I also want to improve the actual encoding, though that's mostly a separate issue and doesn't obviate the above problems with UI design and anything other than video/audio player embeds.

While Safari on macOS now plays WebM video with suitable hardware VP9 codec, iOS Safari and web views have limited support:
 * does take fMP4/VP9 in an HTTP Live Stream video using the native HLS playback
 * does NOT take WebM/VP9 files directly in
 * does take WebM/VP9 via Media Source Extensions (iPad only, requires JS)

However this is different from macOS Safari:
 * does NOT take fMP4/VP9 in an HTTP Live Stream video using the native HLS playback
 * does take WebM/VP9 files directly in
 * does take WebM/VP9 via Media Source Extensions (requires JS)

Additionally note that audio is an issue: either MP3 (but not in fMP4, so complicates setup) or fMP4/AAC (which _is probably_ fine based on Linux distro behavior by Red Hat but may require a legal review).

Tests and issues collection
todo: organize and list details


 * https://phabricator.wikimedia.org/T68722 <- bug w/ several examples of broken things :D
 * double-check if old revisions and previews need to be handled with rendered Graphs instances :D
 * double-check w/ legal on AAC (and consider the MP3 alternative)