User:Brooke Vibber (WMF)/Mobile video playback 2023

Media playback, and in particular video playback, is broken or partially broken in the mobile apps and can be improved. I'm embedding with the combined mobile team's sync meetings to plan some work to fix this; currently laying out the possible things we can do so we can decide on best trade-off.

Basic issues

• complete lack of native WebM and Ogg support in <video> and <audio> on iOS web views
• bare <video> and <audio> on Android may not pick the best resolution
• bare links to Ogg or Opus audio source files are common in pronunciation templates, so even though we produce an MP3 transcode it's not used in this circumstance on iOS unless the frontend JS knows to look for it

todo:

• (in code review) add iOS-compatible VP9-in-HLS streaming video output
• test TimedMediaHandler video and audio output in the apps once HLS updates have gone out for better iOS compatibility
• develop a link-handler for bare audio links (separately for web and for apps? should be small JS...)
• double-check 3d model behavior (isolated in files like video/audio, but different thumbnail output)
• double-check Graphs behavior (not isolated in files, may require different handling)

First-line improvements to video

Currently in code review updates to generate HLS-compatible, MSE-compatible adaptive streaming tracks and .m3u8 playlists. These run natively on current iOS devices, with codecs we already use (VP9 video, MP3 audio) and also can run in other browsers (with Opus audio track).

Short term plan

Short term plan is to enable these for iOS only (preferring the WebM tracks where supported), using the native player support on iOS. Older iOS devices, or Macs that deliberately choose the HLS view, will see a low-resolution MJPEG version.

Long term plan

Using the same tracks in other browsers will allow us to remove most of the high-res WebM VP8 and VP9 transcodes we produce now, since we can use a single streaming VP9-in-MP4 track for all browsers. This currently works via video.js's http-streaming plugin in all browsers *except* Safari, which needs an upstream fix to load the MP3 audio tracks into MSE (Safari doesn't understand Opus in MP4 yet).

Test plan (summer 2023

Currently planning to roll this out to test.wikipedia.org for a shakedown period during which we'll run more experiments on the Android and iOS mobile app experiences with "real data".

Extension:TimedMediaHandler/Test media

App-side possibilities

With the basic format issue taken care of, there's still a need for a good configurable player experience.

iframe embed

Have in-app JS replace the bare players with <iframe>s pointing to the embedded version in the web UI (with ?embedplayer=yes)

• good: this will load the necessary player UI for subtitles, etc
• good: cross-platform
• bad: won't work offline (may be fine?)
• good: loose coupling, the embedplayer=yes param could be reused on other file types
  • bad: however this is not currently implemented. probably deserves first-class support in MediaWiki.
• bad: the same system may not work as well for things like Graphs that don't have a separate page where they live; this needs further investigation
• bad: no app customization of the UI inside the iframe, unless we devise an API that communicates over postMessage or query parameters

Load ResourceLoader modules into the app

Load the video.js player UI, including its ogv.js compat shim on iOS, into the web view from ResourceLoader, bundling it for offline caching

• bad: may require retooling the player setup functions to work in the mobile app environment
• ?: not sure if we are currently doing the 'pull a module from ResourceLoader and save it' sensibly or not :D
• bad: sounds like tight coupling, specific to TimedMediaHandler
  • good: could be future-proofed to be more generic, exposing a specific list of modules that are safe to load outside mediawiki and needed for a given page
• good: app could customize the player a bit via styles or plug-in JS

Native app player

Parse the sources lists from the <video> or <audio> and present the selected resolution/codec via native code

• High-level players built into iOS should take the new HLS tracks once they're available, as well as flat .mp3 for audio transcodes
  • should be no need to worry about WebM compat now
• Exoplayer library on Android should also handle the HLS tracks, as well as the existing flat files
• bad: doesn't help for other types of media that need to ship "behaviors" (code) with their HTML, such as Graphs
• good: complete app control over the player UI

Short term: choices

• is offline important?
  • if no, existing plans are functional and future-proof
  • if yes, can't use iframe.
    • video: need to the download files & strip the playlist to just formats copied (eg, don't pre-download 4K video unless asked to)
    • other types: may have to download blobs of HTML+JS? is this even feasible unless planned for?
• is Graphs important now?
  • if so, probably need a non-iframe JS module
  • alternately, ability to put code in iframe and inject data from the host document (consider security)

Tests and issues collection

todo: organize and list details

• https://phabricator.wikimedia.org/T68722 <- bug w/ several examples of broken things :D
• double-check if old revisions and previews need to be handled with rendered Graphs instances :D
• double-check w/ legal on AAC (and consider the MP3 alternative)