ResourceLoader/Version 1 Design Specification

Tracked in Phabricator
Task T26415

Use-cases

This specification aims to be driven by, and define a prioritization for, a set of use-cases, each of which present their own unique requirements and challenges. The following use cases have been listed in highest to lowest priority.

Core features
Progressive enhancement for views such as the edit or history pages, or skins such as Vector or Monobook. As a part of MediaWiki core, it's critical that the way this system is interacted with complements existing coding conventions and design patterns.
Extensions
Primary functionality as well as progressive enhancement for a wide variety of features provided by extensions, such as editing or discussion systems. The optional nature of extensions presents unique complexities in cases where more than one extension might provide similar or identical client-side resources such as jQuery plug-ins, which can potentially conflict when loaded on the client.
Third-party sites
Delivery of resources for the purpose of allowing access to multimedia content on other sites.
Site customizations
Administrator-authored, wiki-hosted, all-user accessible software packages which modify pages after loaded on the client.
Gadgets
User-authored, wiki-hosted, any-user accessible software packages which modify pages after being loaded on the client.
User-scripts
User-authored, wiki-hosted, single-user-accessible software packages which modify pages after being loaded on the client.

Overview

This library is a combination of a server-side resource delivery service and a client-side resource acquisition library. The primary objective of these two systems is to work together to deliver collections of resources on demand as quickly as possible.

Modules

A module represents a collection of resources. Types of resources are JavaScript, CSS and internationalized messages. For most modules the JS and CSS comes from files; for certain special modules, the JS and CSS is generated by PHP code.

Registration

Modules are registered on the server at start-up. While most modules are representations of collections of files, other modules also exist which provide special behaviors such as generating JavaScript or CSS code or pulling code from a database. Each specific module type shares a parent class of ResourceLoaderModule.

File Module

ResourceLoaderFileModule is initialized with an array which provides information about the locations of the following kinds of resources.

scripts JavaScript file(s)
languageScripts List of JavaScript file(s) keyed by language code
skinScripts List of JavaScript file(s) keyed by skin name
debugScripts JavaScript file(s) to include only when in debug mode
dependencies Name(s) of modules which are to be loaded prior to loading the module
loaderScripts JavaScript file(s) providing manual registration for the module
styles CSS file(s)
skinStyles List of CSS file(s) keyed by skin name
messages List of message keys used by the script

StartUp Module

ResourceLoaderStartUpModule provides a generated dependency map, early browser compatibility detection and code needed to proceed loading when run on a compatible client.

Site Module

ResourceLoaderSiteModule abstracts site-wide JavaScript and CSS resources stored in the database such as MediaWiki:Common.js.

User Module

ResourceLoaderUserModule abstracts per-user JavaScript and CSS resources stored in the database such as User:[UserName]/Common.js.

User Options Module

ResourceLoaderUserOptionsModule abstracts per-user JavaScript and CSS resources generated from user options.

Requesting

Requests are either generated during HTML generation on the server, or dynamically after the document has been loaded on the client.

Server-generated requests are created by the OutputPage class, which collects a list of requested modules and uses them to add JavaScript and CSS include elements to the HTML document after content generation is complete.

Client-generated requests are created by the mediaWiki.loader object, which responds to requests for modules and their dependencies on-demand by adding JavaScript include elements to the document.

When a page is generated on the server, several scripts are added to the document.

Script for registering all modules available on the server with the client, perform a basic browser compatibility check, and inject a script which requests a base package including jquery and mediawiki as scripts only.
Script for setting global variables
Scripts requested by calling either OutputPage::addModuleScripts() or OutputPage::addModuleMessages()
Script for requesting the client loader acquire any modules requested by calling OutputPage::addModules()
Script to call mediaWiki.loader.go(), which will cause the client loader to acquire all requested modules and their dependencies.
Scripts from the ResourceLoaderSiteModule
Scripts from the ResourceLoaderUserModule
Scripts from the ResourceLoaderUserOptionsModule

Delivery

Any number of modules can be obtained through a single HTTP request by passing a list of module names as the value for the modules parameter. Modules names are delimited by a | (pipe) character.

For static requests made in the initial load, the only parameter is used, which delivers partial content. This form of delivery is limited in functionality, and it is the responsibility of the caller to understand the side-effects of using this mode. The only parameter will accept any of the following values.

scripts JavaScript code for scripts, skin scripts, language scripts and debug scripts (if applicable) with an additional piece of code which registers the modules with mediaWiki.loader
styles CSS code for styles and skin-styles
messages JavaScript code which adds messages to mediaWiki.msg

The client loader will only understand a module to be ready if the script for that module and all of its dependencies has been loaded. While it’s possible to load the styles or messages of a module in advance, and then load the whole module later, thus effectively loading the styles or messages twice, the additional information being sent to the client will have no effect other than the additional time spent transferring data since multiple identical CSS style rules are ignored by web browsers and messages are stored by key, preventing any possible duplication.

Startup

The first module loaded is startup which contains a registration function which when executed will register all modules available from the server with the client loader, evaluate if the client will be capable of using the jQuery and MediaWiki libraries, and if so adds a script element to the body which requests them from the resource loader using the only=scripts parameter. This module changes frequently, relies on extremely basic JavaScript support and is relatively small.

The start-up module is in-lined in a <script> tag in the page HTML when the server generates the page. On wikis that cache static HTML pages, ESI (for reverse proxy caches such as Squid and Varnish) or some yet-to-be-determined magic (for $wgFileCache) will be used to in-line an up-to-date version of the start-up module. When using ESI, the start-up module will be requested from the resource loader using an HTTP request, which will have a 5-minute cache expiry as opposed to the far-future expiries set on normal modules.

jquery DOM manipulation library
mediawiki Loader, configuration, basic localization and registration code for all modules that exist on the server

Because the jquery and mediawiki modules are loaded on all pages and change infrequently, loading them discreetly improves their cacheability. Additionally, to support loading all additionally requested modules in one request while calculating their dependencies on the client, the jquery and mediawiki modules must be loaded discreetly.

Once the jquery and mediawiki modules are loaded on the client, mediawiki.loader looks for the registration function created by startup, and if found proceeds with executing and deleting it. Now all modules supported by the server are registered with the mediawiki.loader system.

Additional inline scripts added to the document during HTML generation make calls to mediawiki.loader.load or mediawiki.loader.using to ensure modules are present before the document is considered ready by the client.

A final inline script calls mediawiki.loader.go which works the queue a final time and requests all modules by injecting a script tag at the end of the body.

Packaging

For modules which are loaded secondarily, all resources of the module are packaged together in an implementation call. This packaging wraps all JavaScript code in a closure, encodes all CSS style rules as a string, and encodes all localized messages into a JSON object. This closure, string and object are then passed through the mediaWiki.loader.implement function, which stores the contents of the module on the client. If and when all of the modules’ needs have been met, the messages are made accessible through mediaWiki.msg.get, the CSS is added to the document through appending a style element to the head, and the closure is executed. Upon execution, any modules which were dependent on the recently executed module are reconsidered for immediate execution by evaluating their remaining needs.

URL parameterization

Information about the state of the client is passed through the URL of a resource request. This allows basic resource inclusion logic to take place on the server. The following parameters may be optionally included in a resource request's URL.

lang Name of current user interface language (en, en-gb, zh-cn, etc.)

Affects which language scripts are included with the module if any were registered for that language Defaults to the name of the default language for the site

dir Overrides writing direction (ltr or rtl)

Affects if style properties are flipped horizontally or not Defaults to current language's writing direction

skin Name of current skin (vector, monobook, modern, etc.)

Affects which skin scripts and styles are included with the module if any were registered for that skin Defaults to the name of the default skin for the site

debug Whether scripts should be sent in debug mode

Affect whether debug scripts are included with the module if any were registered Defaults to false

While encoding state information into the URL significantly increases the number of unique requests thus increasing the cache size, it also moves enough inclusion logic to the server to significantly reduce the size of both initial payload and number of secondary requests.

Dependencies

While the order of execution is critical when resolving dependencies, the order in which they are listed and subsequently loaded within a request is not. Modules are registered with the loading system when received by the client, and executed by the loading system when all dependencies have already been met. This is accomplished by automatically wrapping each module's script content in a closure which can be executed independently of the client receiving and parsing the module code.

Resource transformation

Minification

JavaScript and CSS are by default minified before being sent to the client. Minification will be accomplished by stripping whitespace and comments from the code. Future work in this area may include integration of advanced optimizers such as Google's Closure Compiler.

Directional conversion

For languages which use a right-to-left writing direction, style properties can be automatically converted, requiring that only the left-to-right version be written and maintained. Individual style rules or properties can be preserved by preceding them with a /* @noflip */ comment. This processing step can be manually turned on and off by adding a URL parameter to the load request called dir with a value of rtl to turn it on or ltr to turn it off. For on-wiki CSS such as user and site css, flipping is only performed if the requested language has a different direction than the content language.

URL remapping

CSS URLs will be transformed to fix the fact that they are not being included from the same URL they would be directly accessed by
Possibly add the feature of mapping to a CDN

Data-URL embedding

CSS properties that use url() to include images could be optionally marked with a preceding comment such as /* @embed */ which would indicate that the data of the image the URL links to should be encoded into base64 and inserted. To support legacy browsers (IE 7 and below don’t support data URLs), the original rule will be appended with !ie which will be ignored by browsers which support data URLs. Also, because IE 8 doesn’t support data URLs longer than 32 KB, files over 24 KB (which would expand to over 32 KB in base64) are not embedded this way.

Localization

Each module can provide a list of message keys which should be translated and sent to the client when loading the module. The translated messages are built into blobs of text in JSON format and cached in a database indexed by the module's name. A reverse look-up table is also automatically maintained, which is used to detect when a module's blob needs to be rebuilt due to a change in one of its constituent messages.

On the client side, a function^{[the name of the function?]} exists that modules can use to get the content of a message with parameter replacement and parsing of language features like {{PLURAL:}}.

Debugging

Many transformations have the side-effect of obscuring the source code, which can interfere with debugging. The server and client software can be set to run in debug mode, which disables these features. In debug mode all scripts are loaded through individual requests and most transformations are turned off, and the otherwise no-op function mediaWiki.log outputs content to a console if one exists. If a console is not open or is not supported by the client, HTML will be appended to the body which will display console output.

Scalability considerations

Caching and Versioning

Responses served by the resource loader will set Cache-Control headers such that the responses are cached in Squid for 30 days. When the Squid cache entry expires, Squid will re-request the resource with an If-Modified-Since header and get a 304 response back if the resource hasn't changed. On the client side, responses are also cached for 30 days. Client-side cache invalidation is done with style versions, much like in the current system. However, the style version will be the highest modification timestamp of any file registered as part of a module or most recent message cache update, rounded down to a multiple of 10 seconds and formatted in ISO 8601. Calculating this style version may be a relatively expensive operation, so we may need to come up with some caching smartness for it.

The resource loader itself also caches transformed versions of resources it generates, keyed by the MD5 hash of the transformed content.

CSS image links are checked against local files and modified to include their last modified timestamp as a style version.