User:Brion Vibber (WMF)/Mobile video playback 2023

Media playback, and in particular video playback, is broken or partially broken in the mobile apps and can be improved. I'm embedding with the combined mobile team's sync meetings to plan some work to fix this; currently laying out the possible things we can do so we can decide on best trade-off.

Basic issues

 * complete lack of native WebM and Ogg support in and on iOS web views
 * bare and on Android may not pick the best resolution
 * bare links to Ogg or Opus audio source files are common in pronunciation templates, so even though we produce an MP3 transcode it's not used in this circumstance on iOS unless the frontend JS knows to look for it

todo:
 * (in code review) add iOS-compatible VP9-in-HLS streaming video output
 * test TimedMediaHandler video and audio output in the apps once HLS updates have gone out for better iOS compatibility
 * develop a link-handler for bare audio links (separately for web and for apps? should be small JS...)
 * double-check 3d model behavior (isolated in files like video/audio, but different thumbnail output)
 * double-check Graphs behavior (not isolated in files, may require different handling)

First-line improvements to video
Currently in code review updates to generate HLS-compatible, MSE-compatible adaptive streaming tracks and .m3u8 playlists. These run natively on current iOS devices, with codecs we already use (VP9 video, MP3 audio) and also can run in other browsers (with Opus audio track).

Short term plan
Short term plan is to enable these for iOS only (preferring the WebM tracks where supported), using the native player support on iOS. Older iOS devices, or Macs that deliberately choose the HLS view, will see a low-resolution MJPEG version.

Long term plan
Using the same tracks in other browsers will allow us to remove most of the high-res WebM VP8 and VP9 transcodes we produce now, since we can use a single streaming VP9-in-MP4 track for all browsers. This currently works via video.js's http-streaming plugin in all browsers *except* Safari, which needs an upstream fix to load the MP3 audio tracks into MSE (Safari doesn't understand Opus in MP4 yet).

Test plan (summer 2023
Currently planning to roll this out to test.wikipedia.org for a shakedown period during which we'll run more experiments on the Android and iOS mobile app experiences with "real data".

Details to come.

App-side possibilities
With the basic format issue taken care of, there's still a need for a good configurable player experience.

iframe embed
Have in-app JS replace the bare players with s pointing to the embedded version in the web UI


 * good: this will load the necessary player UI for subtitles, etc
 * good: cross-platform
 * bad: won't work offline (may be fine?)
 * good: loose coupling, the  param could be reused on other file types
 * bad: however this is not currently implemented. probably deserves first-class support in MediaWiki.
 * bad: the same system may not work as well for things like Graphs that don't have a separate page where they live; this needs further investigation
 * bad: no app customization of the UI inside the iframe, unless we devise an API that communicates over postMessage or query parameters

Load ResourceLoader modules into the app
Load the video.js player UI, including its ogv.js compat shim on iOS, into the web view from ResourceLoader, bundling it for offline caching


 * bad: may require retooling the player setup functions to work in the mobile app environment
 * ?: not sure if we are currently doing the 'pull a module from ResourceLoader and save it' sensibly or not :D
 * bad: sounds like tight coupling, specific to TimedMediaHandler
 * good: could be future-proofed to be more generic, exposing a specific list of modules that are safe to load outside mediawiki and needed for a given page
 * good: app could customize the player a bit via styles or plug-in JS

Native app player
Parse the sources lists from the or and present the selected resolution/codec via native code


 * High-level players built into iOS should take the new HLS tracks once they're available, as well as flat .mp3 for audio transcodes
 * should be no need to worry about WebM compat now
 * Exoplayer library on Android should also handle the HLS tracks, as well as the existing flat files
 * bad: doesn't help for other types of media that need to ship "behaviors" (code) with their HTML, such as Graphs
 * good: complete app control over the player UI

Short term: choices

 * is offline important?
 * if no, existing plans are functional and future-proof
 * if yes, can't use iframe.
 * video: need to the download files & strip the playlist to just formats copied (eg, don't pre-download 4K video unless asked to)
 * other types: may have to download blobs of HTML+JS? is this even feasible unless planned for?
 * is Graphs important now?
 * if so, probably need a non-iframe JS module
 * alternately, ability to put code in iframe and inject data from the host document (consider security)

Short term: re-encoding for iOS video
My current work patch is doing some fixup to video transcode output which will include an experimental 144p QuickTime MJPEG/MP3 output. This is a low resolution because compression with motion-JPEG is awful, but since JPEG and MP3 are things we already support, and QuickTime/MP4/ISO BMFF is 'standard enough' and widely implemented, it works within our existing rules and more importantly *works on Safari* on all iOS and Mac devices, even the old ones without newer VP9 hardware codecs available, and even when our JS doesn't run to load the WebAssembly decoder shim.

If all goes well I hope to start a batch job to roll out fallback files for all existing uploads and they'll just 'start working'. Other improvements we make in future will give higher res videos, or better interaction, or both.

Long term: re-encoding for iOS video
I also want to improve the actual encoding, though that's mostly a separate issue and doesn't obviate the above problems with UI design and anything other than video/audio player embeds.

While Safari on macOS now plays WebM video with suitable hardware VP9 codec, iOS Safari and web views have limited support:
 * does take fMP4/VP9 in an HTTP Live Stream video using the native HLS playback
 * does NOT take WebM/VP9 files directly in
 * does take WebM/VP9 via Media Source Extensions (iPad only, requires JS)

However this is different from macOS Safari:
 * does NOT take fMP4/VP9 in an HTTP Live Stream video using the native HLS playback
 * does take WebM/VP9 files directly in
 * does take WebM/VP9 via Media Source Extensions (requires JS)

Additionally note that audio is an issue: either MP3 (but not in fMP4, so complicates setup) or fMP4/AAC (which _is probably_ fine based on Linux distro behavior by Red Hat but may require a legal review).

Tests and issues collection
todo: organize and list details


 * https://phabricator.wikimedia.org/T68722 <- bug w/ several examples of broken things :D
 * double-check if old revisions and previews need to be handled with rendered Graphs instances :D
 * double-check w/ legal on AAC (and consider the MP3 alternative)