ResourceLoader/Version 1 Design Specification

'''This document specifies areas of the Resource Loader which have specific design requirements. Please do not make changes to this page unless they correspond to changes in the software.'''

Parts of the document marked in red are parts which have either not been finalized or have yet to be implemented. in either case these parts of the specification are most likely going to be adjusted as implementation details emerge. Discussion about already or not-yet presented requirements and/or feature requests should take place on the talk page.

While collecting and interpreting requirements and feature requests from the Wikimedia Foundation staff and developer community some of these requirements were understood to be either mutually exclusive or out of scope of the resources dedicated to this project or the scale of the system. For this reason, this specification is the result of negotiation between multiple parties.

Use-cases
This specification aims to be driven by, and define a prioritization for, a set of use-cases, each of which present their own unique requirements and challenges. The following use cases have been listed in highest to lowest priority.


 * Core features Progressive enhancement for views such as the edit or history pages, or skins such as Vector or Monobook. As a part of MediaWiki core, it's critical that the way this system is interacted with complements existing coding conventions and design patterns.
 * Extensions Primary functionality as well as progressive enhancement for a wide variety of features provided by extensions, such as editing or discussion systems. The optional nature of extensions presents unique complexities in cases where more than one extension might provide similar or identical client-side resources such as jQuery plug-ins, which can potentially conflict when loaded on the client.
 * Third-party sites Delivery of resources for the purpose of allowing access to multimedia content on other sites.
 * Site customizations Administrator-authored, wiki-hosted, all-user accessible software packages which modify pages after loaded on the client.
 * Gadgets User-authored, wiki-hosted, any-user accessible software packages which modify pages after being loaded on the client.
 * User-scripts User-authored, wiki-hosted, single-user-accessible software packages which modify pages after being loaded on the client.

Overview
This library is a combination of a server-side resource delivery service and a client-side resource acquisition library. The primary objective of these two systems is to work together to deliver collections of resources on demand as quickly as possible.



Modules
A module represents a collection of resources. Types of resources are JavaScript, CSS and internationalized messages. For most modules the JS and CSS comes from files; for certain special modules, the JS and CSS is generated by PHP code.

Registration
Modules are registered on the server at start-up. While most modules are representations of collections of files, other modules also exist which provide special behaviors such as generating JavaScript or CSS code or pulling code from a database. Each specific module type shares a parent class of ResourceLoaderModule.

File Module
ResourceLoaderFileModule is initialized with an array which provides information about the locations of the following kinds of resources.


 * scripts JavaScript file(s)
 * languageScripts List of JavaScript file(s) keyed by language code
 * skinScripts List of JavaScript file(s) keyed by skin name
 * debugScripts JavaScript file(s) to include only when in debug mode
 * dependencies Name(s) of modules which are to be loaded prior to loading the module
 * loaderScripts JavaScript file(s) providing manual registration for the module
 * styles CSS file(s)
 * skinStyles List of CSS file(s) keyed by skin name
 * messages</tt> List of message keys used by the script

StartUp Module
ResourceLoaderStartUpModule provides a generated dependency map, early browser compatibility detection and code needed to proceed loading when run on a compatible client.

Site Module
ResourceLoaderSiteModule abstracts site-wide JavaScript and CSS resources stored in the database such as MediaWiki:Common.js.

User Module
ResourceLoaderUserModule abstracts per-user JavaScript and CSS resources stored in the database such as User:[UserName]/Common.js.

User Options Module
ResourceLoaderUserOptionsModule abstracts per-user JavaScript and CSS resources generated from user options.

Requesting
Requests are either generated during HTML generation on the server, or dynamically after the document has been loaded on the client.

Server-generated requests are created by the OutputPage class, which collects a list of requested modules and uses them to add JavaScript and CSS include elements to the HTML document after content generation is complete.

Client-generated requests are created by the mediaWiki.loader object, which responds to requests for modules and their dependencies on-demand by adding JavaScript include elements to the document.

When a page is generated on the server, several scripts are added to the document.


 * 1) Script for registering all modules available on the server with the client, perform a basic browser compatibility check, and inject a script which requests a base package including jquery and mediawiki as scripts only.
 * 2) Script for setting global variables
 * 3) Scripts requested by calling either OutputPage::addModuleScripts or OutputPage::addModuleMessages
 * 4) Script for requesting the client loader acquire any modules requested by calling OutputPage::addModules
 * 5) Script to call mediaWiki.loader.go, which will cause the client loader to acquire all requested modules and their dependencies.
 * 6) Scripts from the ResourceLoaderSiteModule
 * 7) Scripts from the ResourceLoaderUserModule
 * 8) Scripts from the ResourceLoaderUserOptionsModule

Delivery
Any number of modules can be obtained through a single HTTP request by passing a list of module names as the value for the modules parameter. Modules names are delimited by a | (pipe) character.

For static requests made in the initial load, the only parameter is used, which delivers partial content. This form of delivery is limited in functionality, and it is the responsibility of the caller to understand the side-effects of using this mode. The only parameter will accept any of the following values.


 * scripts</tt> JavaScript code for scripts, skin scripts, language scripts and debug scripts (if applicable) with an additional piece of code which registers the modules with mediaWiki.loader
 * styles</tt> CSS code for styles and skin-styles
 * messages</tt> JavaScript code which adds messages to mediaWiki.msg

The client loader will only understand a module to be ready if the script for that module and all of its dependencies has been loaded. While it’s possible to load the styles or messages of a module in advance, and then load the whole module later, thus effectively loading the styles or messages twice, the additional information being sent to the client will have no effect other than the additional time spent transferring data since multiple identical CSS style rules are ignored by web browsers and messages are stored by key, preventing any possible duplication.

Startup
The first module loaded is startup which contains a registration function which when executed will register all modules available from the server with the client loader, evaluate if the client will be capable of using the jQuery and MediaWiki libraries, and if so adds a script element to the body which requests them from the resource loader using the only=scripts parameter. This module changes frequently, relies on extremely basic JavaScript support and is relatively small.

The start-up module is in-lined in a tag in the page HTML when the server generates the page. On wikis that cache static HTML pages, ESI (for reverse proxy caches such as Squid and Varnish) or some yet-to-be-determined magic (for $wgFileCache) will be used to in-line an up-to-date version of the start-up module. When using ESI, the start-up module will be requested from the resource loader using an HTTP request, which will have a 5-minute cache expiry as opposed to the far-future expiries set on normal modules.


 * jquery</tt> DOM manipulation library
 * mediawiki</tt> Loader, configuration, basic localization and registration code for all modules that exist on the server

Because the jquery and mediawiki modules are loaded on all pages and change infrequently, loading them discreetly improves their cacheability. Additionally, to support loading all additionally requested modules in one request while calculating their dependencies on the client, the jquery and mediawiki modules must be loaded discreetly.

Once the jquery and mediawiki modules are loaded on the client, mediawiki.loader looks for the registration function created by startup, and if found proceeds with executing and deleting it. Now all modules supported by the server are registered with the mediawiki.loader system.

Additional inline scripts added to the document during HTML generation make calls to mediawiki.loader.load or mediawiki.loader.using to ensure modules are present before the document is considered ready by the client.

A final inline script calls mediawiki.loader.go which works the queue a final time and requests all modules by injecting a script tag at the end of the body.

Packaging
For modules which are loaded secondarily, all resources of the module are packaged together in an implementation call. This packaging wraps all JavaScript code in a closure, encodes all CSS style rules as a string, and encodes all localized messages into a JSON object. This closure, string and object are then passed through the mediaWiki.loader.implement function, which stores the contents of the module on the client. If and when all of the modules’ needs have been met, the messages are made accessible through mediaWiki.msg.get, the CSS is added to the document through appending a style element to the head, and the closure is executed. Upon execution, any modules which were dependent on the recently executed module are reconsidered for immediate execution by evaluating their remaining needs.

URL parameterization
Information about the state of the client is passed through the URL of a resource request. This allows basic resource inclusion logic to take place on the server. The following parameters may be optionally included in a resource request's URL.

Affects which language scripts are included with the module if any were registered for that language Defaults to the name of the default language for the site Affects if style properties are flipped horizontally or not Defaults to current language's writing direction Affects which skin scripts and styles are included with the module if any were registered for that skin Defaults to the name of the default skin for the site Affect whether debug scripts are included with the module if any were registered Defaults to false
 * lang</tt> Name of current user interface language (en, en-gb, zh-cn, etc.)
 * dir</tt> Overrides writing direction (ltr or rtl)
 * skin</tt> Name of current skin (vector, monobook, modern, etc.)
 * debug</tt> Whether scripts should be sent in debug mode

While encoding state information into the URL significantly increases the number of unique requests thus increasing the cache size, it also moves enough inclusion logic to the server to significantly reduce the size of both initial payload and number of secondary requests.

Dependencies
While the order of execution is critical when resolving dependencies, the order in which they are listed and subsequently loaded within a request is not. Modules are registered with the loading system when received by the client, and executed by the loading system when all dependencies have already been met. This is accomplished by automatically wrapping each module's script content in a closure which can be executed independently of the client receiving and parsing the module code.

Minification
JavaScript and CSS are by default minified before being sent to the client. Minification will be accomplished by stripping whitespace and comments from the code. Future work in this area may include integration of advanced optimizers such as Google's Closure Compiler.

Directional conversion
For languages which use a right-to-left writing direction, style properties can be automatically converted, requiring that only the left-to-right version be written and maintained. Individual style rules or properties can be preserved by preceding them with a /* @noflip */</tt> comment. This processing step can be manually turned on and off by adding a URL parameter to the load request called dir with a value of rtl to turn it on or ltr to turn it off. For on-wiki CSS such as user and site css, flipping is only performed if the requested language has a different direction than the content language.

URL remapping

 * CSS URLs will be transformed to fix the fact that they are not being included from the same URL they would be directly accessed by
 * Possibly add the feature of mapping to a CDN

Data-URL embedding
CSS properties that use url</tt> to include images could be optionally marked with a preceding comment such as /* @embed */</tt> which would indicate that the data of the image the URL links to should be encoded into base64 and inserted. To support legacy browsers (IE 7 and below don’t support data URLs), the original rule will be appended with !ie which will be ignored by browsers which support data URLs. Also, because IE 8 doesn’t support data URLs longer than 32 KB, files over 24 KB (which would expand to over 32 KB in base64) are not embedded this way.

Localization
Each module can provide a list of message keys which should be translated and sent to the client when loading the module. The translated messages are built into blobs of text in JSON format and cached in a database indexed by the module's name. A reverse look-up table is also automatically maintained, which is used to detect when a module's blob needs to be rebuilt due to a change in one of its constituent messages.

On the client side, a function exists that modules can use to get the content of a message with parameter replacement and parsing of language features like NaN undefineds.

Debugging
Many transformations have the side-effect of obscuring the source code, which can interfere with debugging. The server and client software can be set to run in debug mode, which disables these features. In debug mode all scripts are loaded through individual requests and most transformations are turned off, and the otherwise no-op function mediaWiki.log</tt> outputs content to a console if one exists. If a console is not open or is not supported by the client, HTML will be appended to the body which will display console output.

Caching and Versioning
Responses served by the resource loader will set Cache-Control headers such that the responses are cached in Squid for 30 days. When the Squid cache entry expires, Squid will re-request the resource with an If-Modified-Since header and get a 304 response back if the resource hasn't changed. On the client side, responses are also cached for 30 days. Client-side cache invalidation is done with style versions, much like in the current system. However, the style version will be the highest modification timestamp of any file registered as part of a module or most recent message cache update, rounded down to a multiple of 10 seconds and formatted in ISO 8601. Calculating this style version may be a relatively expensive operation, so we may need to come up with some caching smartness for it.

The resource loader itself also caches transformed versions of resources it generates, keyed by the MD5 hash of the transformed content.

CSS image links are checked against local files and modified to include their last modified timestamp as a style version.