ResourceLoader/Architecture

This page summarizes the many Features of ResourceLoader which together build the rich experience and front-end performance.

Module properties

 * Code resources (scripts and styles)
 * Messages
 * Dependencies
 * Group

Conditions
Scripts can be conditionally included in a module based on the current production mode, selected skin and user language. This keeps responses as small as possible by only including relevant components while also allowing progressive enhancement and conditional inclusion of features based on the environment.

Such as the  method, which is just a no-op closure in production mode, whereas in debug mode the   module loads additional components that implement a visual console for browsers that don't have one by default.

Embedding
In order to reduce the number of HTTP requests for images used in the interface, ResourceLoader makes use of Data URI embedding. When enabled, images will be automatically base64-encoded and embedded into the stylesheet in-place. While it will make the stylesheet larger, it improves performance by removing the overhead associated with requesting all those additional files. The actual server response (which contains the minified result of all concatenated stylesheets and all embedded images in a single request) uses gzip compression. This enables the response to function a bit like a "super sprite" (more on this below). Regardless of the expansion caused by base64 encoding, the ResourceLoader response is smaller in size than the sum of the individual binary files.

To trigger the embedding the " " annotation is used in a CSS comment. For example:

Another interesting point is that using this technique makes traditional sprites obsolete. The motivation of a sprite is good, it:
 * Reduces HTTP requests by combining multiple images into one.
 * Increases power of compression by providing more sample data in a single file.

But sprites come with a few caveats:
 * Maintenance (if an image needs to be updated, one has to regenerate the entire sprite, update the background positions, etc.)
 * Produces overly complex CSS (unrelated stylesheets would be referencing the same image with arbitrary background positions, it isn't obvious which image is being referred to, like " " vs. " )".
 * ResourceLoader Wikimania 2011.pdf  Imposes restrictions on usage of the image (no freedom in background-repeat, -size or -position as that would "leak" other parts of the sprite).

These caveats aren't the end of the world (sprites are in wide use, clearly they do work). And many front-end systems use sprites, even in a semi-automated fashion (making some of the above caveats less painful). But, using the automated embedding technique: the above caveats don't apply: .. the advantages of sprites also hold up: ... and, in addition to that, it gets even more out of it:
 * No maintenance
 * Clean CSS
 * No restrictions
 * Reduced number of HTTP requests.
 * Increased power of compression
 * No download delay: Once the stylesheet is there, all the images are there as well.
 * When using an interactive state (e.g. ) browsers only start the download once the relevant state is active. By embedding it there will be no flash during the download, as the image is instantly available.
 * The number of requests is even smaller. The CSS and the images are now in the same request.
 * The Gzip compression rate will be even higher (completely unrelated modules can be concatenated and compressed together, without having to worry about anything. Also, this way PNG headers can be compressed even stronger).

Flipping

 * For more information about directionality support in MediaWiki, see Directionality support.



With the Flipping functionality it is no longer necessary to manually maintain a copy of the stylesheet for right-to-left languages. Instead ResourceLoader automatically change the direction of direction-dependent CSS rules (and more). Internally, ResourceLoader uses CSSJanus to accomplish this. ResourceLoader comes with a PHP port of CSSJanus (a Python library maintained by Google Inc.). This library provides that smart "flipping" logic.

Aside from flipping direction-oriented values, property-names and shorthand values. It also supports swapping direction specific iconography by using filenames ending in  and    respectively.

Consider the following example:

When put in a file that is loaded by ResourceLoader, without any additional changes or configuration, it will be automatically turned into the following for users having the interface in a right-to-left language:

Sometimes you may wish to exclude a rule from being flipped. For that the  annotation is provided. This instructs CSSJanus to not flip the following CSS declaration, or when used in the selector part, the entire following CSS ruleset.

For example:

Output will be:

Conditions

 * See the section under  for more information

Similar to scripts, the styles property also features the ability to compose the module dynamically based on the environment.

Conditions
As with the other two resource types, the messages are also optimized to load only what is necessary for the requesting environment. Messages will be fetched from MediaWiki's localization framework (including proper fallback to other languages for messages that are not translated yet). And only 1 resulting set of messages is delivered the client, the set that the user will need.

On-demand package generation
ResourceLoader features on-demand generation of the module packages. The on-demand generation is very important in MediaWiki because cache invalidation can come from many places. Here's a few examples:
 * Core
 * Extensions Core and extensions generally only change when a wiki is upgraded. But especially on large sites such as Wikipedia, deployments happen many times a day (even updates to core).
 * Users Wiki users granted certain user rights (administrators by default) have the ability to modify the "site" module (which is empty by default and will be loaded for everybody when non-empty). This is all without servers-side access, these scripts/styles are stored as wiki pages in the database. On top of that, each user also has its own module space that is only loaded for that user.
 * Translators The interface messages are shipped with MediaWiki core and are generally considered part of core (and naturally update when upgrading/deploying core). However wikis can customize their interface by using the MediaWiki message namespace to modify interface messages (or create new ones to use in their own modules).

Every module has as an automatically generated last-modified timestamp. This timestamp is based on a number of factors. All together, used to ensure proper cache busting when needed while also avoiding unnecessary re-generation of the package when it isn't needed.
 * JavaScript files and CSS files Last modified timestamps from the files on disk.
 * Files referenced in CSS Extracted from the stylesheets (e.g., see also ). Last modified timestamps from the files on disk.
 * Localization Last time any of the messages has been updated in the localization storage.
 * Dependencies It works recursively for any modules that this module depends on.

Because of all these different origins and cache invalidation factors, it is not desirable to have to manually (or scheduled) perform a "build" of some kind that would generate a huge package with everything in it. Not only would it waste resources re-building the same module most of the time, it would also be impractical to have one big build with everything in it because there are over 350 modules on an average Wikipedia site. Many of these come from extensions and wiki-users and these modules are likely only needed/loaded on certain pages or in certain states of the interface. For more on that see.

Whenever a module is requested from the server, the above is evaluated. If needed, the cache will be re-generated. All phases of the packaging process have been optimized to be able to run on-demand on the web server. No build scripts, no periodic cron tasks.

JavaScriptMinifier
Although the re-generation of a module package should be relatively rare (since cache is very well controlled), when it does happen it has to perform well from a web server.

For that reason it doesn't use the famous JSMin.php library (based on Douglas Crockford's JSMin) because it is too slow to run on-demand from the web. Although JSMin.php only takes about 1 second for (which is okay if you're on the command-line), when working on-demand from the web (with potentially dozens of large files lined up to be minified) waiting this long is unacceptable. Especially if potentially hundreds of thousands of requests could come in at the same moment, all finding out that the cache isn't up to date, to avoid a cache stampede.

Instead ResourceLoader uses an implementation of Paul Copperman's JavaScriptMinifier, which is up to 4X faster than JSMin. In addition to the speed, time has told that JavaScriptMinifier interprets the JavaScript syntax more correctly and succeeds in situations where JSMin outputs invalid JavaScript. The output size of JavaScriptMinifier is slightly larger than JSMin (about 0.5%; based on a comparison by minifying jquery.js, for which the difference was 0.8KB). The reason this is not considered a loss is because it is put in the bigger picture. ResourceLoader doesn't aim to compress as small as can be no matter the cost. Instead it aims for balance, get large gains in a wide range of areas while also featuring instant cache invalidation, fast module generation, a transparent "build"-free environment for the developer, etc. The fact that it could be a little bit smaller then becomes an acceptable trade off.

JavaScriptMinifier is a standalone library. It is free, open-source and available from the MediaWiki repository ([//svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/libs/JavaScriptMinifier.php?view=markup svn] &bull; git)

Workflow

 * This section is incomplete or outdated.

So how does all this play out in the end? Lets walkthrough a typical page view in MediaWiki, focussing on ResourceLoader-related events.

Client-side Loader

 * Module registry
 * Load only when needed
 * Asynchronous requests
 * Execution separated from loading/parsing

Execution

 * Direct or delayed execution as appropriate based on module dependencies
 * Insert messages and styles into memory before script execution

Balance

 * Batching
 * Groups
 * Alphabetical order
 * Module timestamps in url (to allow static reverse-proxy caching)

Load response
Modules are composed based on the the following environmental variables:
 * mode
 * skin
 * user language

The result of the generated module package is cached (see also ). So that if one client requests module A+B and later another client is requesting B+C, B will not be re-build.

The load responder supports various methods of encouraging client-side caching, such as responding with an empty  to   headers if possible.