Extension:TimedMediaHandler/VP9 transition

I'm planning to start migrating Wikimedia's video playback transcodes from the older WebM VP8 and Vorbis codecs to the newer WebM VP9 and Opus codecs. The transition should start soon, in mid July, once a few remaining issues are resolved.

Big thanks to Wikimedia's Ops team for reassigning some servers that were freed up recently to help with this transition!

Also thanks to Microsoft for adopting the royalty-free, open-source VP9 and Opus codecs in MS Edge; as of the latest Windows 10 update they "just work" out of the box, as they do in Chrome and Firefox.

What does this mean?
The biggest difference will be that the files used for on-wiki video playback will be about 30% smaller, while retaining about the same or better quality. This will reduce bandwidth usage for playback as well as saving disk space on the servers.

The filenames of the actual playback files will change slightly, from ".webm" to ".vp9.webm". Direct links to these transcoded playback files are discouraged; API clients should use videoinfo/derivatives or transcodestatus to get current URLs.

There is no change in support for uploaded file formats; existing original files are not being modified, and every file that works now will continue to work. URLs to existing original files will continue to work.

Encoding times could be 3-4 times longer initially, but this is something we can improve with updated packages for improved threading. Currently evaluating whether that can be put in place before starting transition.

Brower support
Browser support is almost the same as for the previous configuration.

WebM VP9/Opus videos are natively supported by:
 * Chrome and Chromium-based browsers (Opera, Brave, etc)
 * Firefox
 * Edge 17 and up (Windows 10 version 1803)

And are supported via the ogv.js player shim on:
 * Safari (up to 720p24 on fast machines)
 * Edge 16 and below (up to 720p24 on fast machines)
 * Internet Explorer 11 (up to 240p24 on fast machines; requires Flash for audio)

Note that the old Google WebM Components for IE did allow for native VP8/Vorbis playback in IE 11 if you manually installed it, but that package does not support VP9/Opus. If you require high-resolution video playback and are affected by this, the recommended upgrade path is to use another browser such as Chrome or Firefox in place of IE 11.

Server-side changes
Because VP9 encoding is more complex than VP8, the scaled and transcoded output files will take longer to produce than the old VP8 files did, CPU per CPU. To compensate and to help along the transition, some additional machines have been reassigned to handle the specialized job queue for video transcoding. (These were previously running image thumbnailing until they were replaced by the newer thumbor system.)

The end-to-end conversion time for each file will get 3-4x slower per CPU, but overall times should stay about the same as before if we're able to deploy updated libvpx and ffmpeg packages supporting macroblock-row-based multithreading first.

Transition
Newly uploaded files will start producing VP9 output automatically.

Old files will continue to hold onto their VP8 versions, which will continue to play them until they get replaced.

A batch process (cleanupTranscodes.php) will go through the backlog generating new VP9 versions to replace them. This process is throttled to minimize disruption to foreground operations.

Note to bot tool authors: please do not manually reschedule a large number of files for new VP9 transcodes! The batch process throttles itself but anything scheduled via the UI or API will run as soon as possible. Too many will clog up the queue.

The complete transition is expected to take several weeks, with a large margin of error, and should present minimal disruption.

Technical details
Filenames/URLs for old VP8 transcodes end in ".webm"; the new VP9 transcodes end in ".vp9.webm".

For instance this output file: https://upload.wikimedia.org/wikipedia/commons/transcoded/3/3e/Ailurus_fulgens.ogv/Ailurus_fulgens.ogv.480p.webm will be obsoleted by this one (does not yet exist): https://upload.wikimedia.org/wikipedia/commons/transcoded/3/3e/Ailurus_fulgens.ogv/Ailurus_fulgens.ogv.480p.vp9.webm

Video scaler provisioning notes
IIRC the newly set up machines, and also the old machines, have dual 10-core/20-thread processors. In theory this means could crank the threads up to 40 (hyperthreading) for maximum parallelism when lightly loaded, however there are diminishing returns beyond 8-16 threads (for HD) or 2-4 threads (for SD and low resolutions). Due to the way row-mt threading in libvpx works, not all threads will be loaded at all times, so you won't see sustained 1600% CPU load from a 16-thread ffmpeg process (at full HD you'd expect to see more like 800%, lower resolutions more like 400% or 200%). Thus the configuration should overprovision from the point of view of $wgFFmpegThreads * the number of job runners per machine.

Risks and contingency plans
Going forward with VP9 on the current Debian 9.x ffmpeg package would increase encoding time by 3-4x, which may cause timeouts for very long videos (conference talks, full-length films). Prioritizing backport of libvpx 1.7 to mitigate this (allows using all cores, which lets us stretch our legs on the 20-core/40-thread queue runners). Moritz will attempt a backport; TMH patch to allow using this mode once it's available.

The main remaining risk is that we'll uncover new bugs in the encoding process while running batch encodes, which could lead to clogging the queue runners.

Batch process for running the re-encoders is cleanupTranscodes.php which will run on terbium in a screen session, and can be shut down by a root in case of emergency. Shutting down this process will prevent new encoding jobs from being enqueued for old uploads, but won't stop for new uploads.

If something goes awry with the transcoder processes, problems should be isolated to the set of machines running job queue runners for the WebVideoTranscode / WebVideoTranscodePrioritized queues. They can be shut down, reset, whatever as necessary if necessary.

If a major compatibility problem is found that requires rolling back to VP8, contingency plan is roughly:
 * switch $wgEnabledTranscodes back from ENC_VP9_* to ENV_WEBM_* constants
 * create new VP8 .webm files for newly-uploaded files with cleanupTranscodes.php
 * leave the .vp9.webm files where they are or delete them with cleanupTranscodes.php
 * clear out the job queue manually