User:Roan Kattouw (WMF)/ResourceLoader and build steps

History note
ResourceLoader was developed in 2010. It predates modern bundlers written in JavaScript, like Webpack, Rollup, etc. Webpack is sometimes described as "Makefiles for the web", and in fact, we used to use Makefiles for concatenating and minifying JavaScript in 2009-2010. We wrote ResourceLoader because that was a pain. An explicit design goal of ResourceLoader was to avoid needing a build step and to do everything on the fly. This was partly for developer productivity reasons: versioning build output in the repo is a pain, and at the time most front-end engineers didn't know how to use command-line build steps (because that wasn't a thing in front-end land back then).

Because of the integration with MediaWiki, ResourceLoader is written in PHP. Nowadays, software that processes JavaScript code and CSS (like bundlers, transpilers, etc) is almost always written in JavaScript. But in 2010, nodejs was this crazy new idea that only some people had heard of (and npm was only a few months old when we started planning out RL). It seemed reasonable at the time to think that we wouldn't need very advanced transformations, and that we could write them all in PHP (in fact, we ported CSSJanus from Python to PHP). Nowadays, that isn't a reasonable assumption anymore, because transformations have gotten way more advanced, and they're all written in JS.

Transformation
Concatenation, minification, JS+CSS+i18n bundling, LESS compilation, CSSJanus, CSS URL remapping, data URI embedding (not used as much anymore)

Dependency management
Resolve dependencies, execute in the right order, per-file require

Caching
Cache busting using version hashes; versioning based on final content but sometimes pre-transform file content; transformation caching

Varying on skin and language
There isn't just one version of a module: different languages have different i18n messages, some are RTL and some aren't, and we allow for skin-specific JS and CSS.

MediaWiki integration and non-file content
RLWikiModule, RLSkinModule, config vars, user settings, dynamic JSON files, LESS import path (in extensions, without needing to know the relative path to core), i18n messages as LESS vars, user scripts/styles, Gadgets

Module loading and registration
MW integration to gather module registrations across core and extensions, gather the list of modules to load on each page (from core and extensions), allow lazy-loading.

ResourceLoader's strengths and limitations
Strength: integration with MW allows for wiki pages and config as content

Weakness: on-the-fly processing must be fast and be in PHP

Weakness: doesn't analyze JS to do automatic dependency detection / tree shaking; instead have to manually define a module's files and dependencies

Getting a trusted build output
The build output is usually unreviewable. Committing built output to repo is annoying, but also a security risk (building dev's laptop is not a very secure environment, could be compromised, CR wouldn't notice compromised build output). Running build on deployment server is scary because executing arbitrary code from npm, and e.g. webpack could be compromised in npm. Need a controlled build env that runs only trusted code, sandboxed, whose output we can trust. (Not needed for reviewable output.)

Duplicated bundler runtime
Bundlers like webpack are intended to produce the entire bundle, not one of several sub-bundles. If multiple extensions use Webpack to generate their own bundle, then if multiple bundles are loaded, Webpack runtime is loaded twice.

Non-global tree shaking leads to duplication
If two extensions use Webpack to tree-shake the same library, they'll duplicate parts of it.

Debugging is harder
Without source maps, or other special support, you won't be able to see the original (non-compiled) code in the browser debugger, or be able to tell which file it came from. RL has some of these problems already, but in-browser debugging is pretty workable even in non-debug mode thanks to deminification (stack traces are pretty useless in non-debug mode though). Loading a built file through RL makes these issues worse. This is partly solvable by loading files differently on dev setups (from a local nodejs server), but we also need to be able to debug in production.

You're going to use ResourceLoader anyway
For things like CSSJanus, i18n messages. You can't pre-build the actual output in most cases, because it varies by language and skin, and there are too many of those. Your module is probably only loaded on some pages, or it may need to be lazy-loaded in some cases.

File concatenation and management
This is not as good in RL as in Webpack, but fairly reasonable

Tree shaking
Trying to solve a global problem locally doesn't work

The way forward for ResourceLoader
I don't really know. Using modern transformations written in JS would be nice. It would get us ES6 support, TypeScript, pre-compiled Vue templates, better JS minification, better RTL flipping (I believe postcss's RTL plugin is better than CSSJanus), more interesting CSS transformations with postcss, and who knows what else. For many reasons, I think we'll probably want/need to continue to avoid build steps and prefer on-the-fly processing. One way to do this might be to move most of RL into a nodejs service, which talks to a PHP endpoint for some of the MW-specific things it needs. Another way could be to keep most of RL in PHP, but have it call out to JS implementations of the transformation steps, either by shelling out or through the v8js PHP extension. For not-fast processing steps we might need caching with pre-population.