MediaWiki

Build step

This page demystifies what build steps are and how they're used in MediaWiki.

Overview

edit
 
Many problems are only solvable when both ahead-of-time and just-in-time build steps are used.

There are two types of build steps:

  • Ahead-of-time: the traditional concept of a build step comes from compiled languages like C where a program would be "built" through a process of compilation, linking, optimization, and other sub-steps that produced an executable. A build step may mean invoking a fully featured JavaScript bundler like Webpack or just calling SVGO on .svg files. It varies as needed per project.
  • Just-in-time: building on-the-fly is useful when key parameters vary during execution and cannot be known ahead-of-time. In MediaWiki, the most prevalent just-in-time build step is ResourceLoader which provides Less compilation, CSS and JavaScript minification, internationalization, URL mapping, and bundling build steps. This article focuses all just-in-time build step discussions on the ResourceLoader implementation.

Many projects have requirements that cannot be solved at runtime and are only solvable by traditional ahead-of-time build steps. In practice, both ahead-of-time and just-in-time are used for for many projects in MediaWiki out of necessity.

Benefits and limitations

edit

Ahead-of-time

edit
Benefits
edit
  • No build time constraints.
  • No tooling constraints. Industry standards and the best tooling available can be used which are more welcoming to newcomers and more sustainable for the Foundation. Ahead-of-time build steps are a gateway to the entire wealth of tooling in the JavaScript ecosystem second only to package.json itself.
  • Build product outputs do not vary and can be inspected, analyzed, and tested all without standing up a full MediaWiki stack to generate build products.
  • Immense flexibility to leverage any tool in the JavaScript ecosystem. An ahead-of-time build step, formal or otherwise, enables an extraordinarily useful abstraction between the code written by developers and the bytes shipped to users. This valuable separation allows for developers to optimize for both development and performance.
  • Code can easily be librarized and shared in any context within or without MediaWiki. Sharing code is just an npm install away.
  • Significantly greater flexibility to solve problems.
  • Tooling and code are canonical, not MediaWiki-specific. This opens up new development and code reuse possibilities within and without MediaWiki as well as enables us to better meet the expectations of newcomers from non-MediaWiki backgrounds interested in using our code or contributing.
  • Best possible performance in terms of bytes shipped and load time, and CPU usage and page rendering.
  • Provides a seam for any and all automations.
Limitations
edit
  • Build product outputs do not vary. There's flexibility to integrate with ResourceLoader to overcome these limitations, such as picking a locale-optimized version, but there's a lot more history of solving these sorts of problems using ResourceLoader directly.
  • Ahead-of-time compilation has historically not had much continuous integration support at Wikimedia. These processes are being evolved using containerization and PipelineLib. In the interim, asset versioning is clumsy. Repos require extra tooling, most JavaScript patches will create merge conflicts, and Git history is polluted with build products. This friction hurts newcomers especially.
  • Build steps at MediaWiki often rely on JavaScript tooling which have notoriously large dependency trees. However, most are limited to development-only dependencies as opposed to runtime dependencies.

Just-in-time

edit
Benefits
edit
  • The status quo. High cohesion with the current MediaWiki ecosystem which has strong patterns for internationalization.
  • Required when key parameters vary and cannot be known ahead-of-time.
  • Centralized build step in Core. All changes made apply to all projects.
Limitations
edit
  • Every just-in-time build step must be extremely performant, so fast that it can run on-the-fly, or pages will load slowly. Additionally, sequential steps cannot be appended ad infinitum.
  • Effectively, ResourceLoader's just-in-time build steps can only use tools written in PHP. JavaScript execution is not possible. For practical purposes, this means the only tools used at MediaWiki are built by Wikimedia or adopted abandonware which greatly impedes evolution and has a significantly negative impact on the development culture.
  • Just-in-time build steps are less secure. They execute on production servers and serve content directly to the user. This eliminates the separation between development and runtime-only dependency trees, which can dramatically increase the attack surface, sometimes by orders of magnitude. Additionally, build outputs are shipped directly to the user without any opportunity for security review. When it comes to security, a just-in-time build step always strives to be as secure as an ahead-of-time build step that produces static outputs.
  • Just-in-time build steps are custom and complex. An ahead-of-time build step can easily be a one-liner that invokes standard tooling but the equivalent just-in-time build step, if one exists, is just as likely to be hundreds of lines of custom code. Historically, these custom steps have suffered from bus factor and received little attention beyond basic life support. Few engineers possess the abilities to write code of the caliber needed to add new build steps or change existing ones, which means the rest of Wikipedia and WMDE is blocked on their evolution. For example, we have been unable to keep pace with fundamental features like source map support (a formal request since 2013) or ES6 transpilation. In fact, there are laundry lists of missing features now standard elsewhere. The lack of standard functionality means that developing any code at Wikimedia is a completely different and far slower experience than the rest of the industry.
  • Just-in-time build step outputs have worse caching. The most advanced build step executed at runtime endeavors to have the same caching that comes out-of-the-box with an ahead-of-time build step: a plain file on disk.
  • Code reuse and librarization and integration requires significant work.
  • What you write is what you ship. No ES6, TypeScript, or language compilation support. At time of writing, this means ES5 only.
  • The version of Less used must match MediaWiki's version (very old).
  • Non-standard.
  • SVGs must be committed minified or delivered to the client in "debug mode."
  • Very primitive debug and development capabilities. No hot module replacement, automated chunk splitting, or tree-shaking, for example.
  • ResourceLoader's just-in-time build step is unavailable for certain projects such as the Page Content Service and libraries.
  • No clear scope or path forward for present and future needs.

Business and use cases

edit

See the platform evolution section of the annual plan, ahead-of-time build steps are necessary in conjunction with just-in-time build steps to achieve the following outcomes:

  • "The ability to build modern experiences" was no better demonstrated than by the new ContentTranslation app which was developed using Vue CLI (a Webpack wrapper) and generated using an ahead-of-time build step.
  • Supporting efficient and effective Wikibase development is also identified. WMDE's Wikibase is a well-known user of ahead-of-time build steps in multiple projects.
  • Infrastructure to serve "with high performance" demands the best profiling and optimization tooling available, not ancient tools that can only run on-the-fly.
  • Significantly "improving engineering productivity" requires bold changes to how we develop. Not small, back-dated improvements like adding source map support but modern development tooling.
  • All bullets under outcome #2, "Fully automated and continuous code health and deployment infrastructure," and outcome #3 "Tooling for contributors is easy to use, well-documented, and accessible to users, increasing engagement and contribution."

Notable projects using an ahead-of-time build step

edit
  • OOUI: Portions of this library are built with Grunt and a suite of packages from NPM for minification, uglification, and additional processing. The results are dozens of build products that are file-copied into Core manually.
  • Page Previews: All JavaScript sources are transpiled from ES6+ to ES5 with a Webpack-powered ahead-of-time build step.
  • Wikibase : Ahead-of-time build tools are used by Wikibase including Webpack, TypeScript, and a plethora of other standards to serve the Wikidata communities.
  • MultimediaViewer: Ahead-of-time build steps replace any human readable source SVGs with optimized, machine-readable outputs.
  • MediaWiki: Core uses an ahead-of-time build step on every deployment in a process called "a full scap."
  • MobileFrontend: All JavaScript is built by Webpack using an ahead-of-time build step.
  • Wikipedia for KaiOS: uses Webpack using an ahead-of-time build step.
  • ContentTranslation: a fully modern Vue.js interface using Vue CLI and standard tooling executed by an ahead-of-time build step.
  • Wikipedia.org: synchronizing sister project statistics.
  • VisualEditor: demos are built using an ahead-of-time build step.

Ahead-of-time build steps are everywhere in Gruntfiles, Gulpfiles, Webpack configs, NPM package.json files, and shell scripts.

See also

edit
Retrieved from "https://www.mediawiki.org/w/index.php?title=Build_step&oldid=5109615"