Technical Decision Forum/Published Decision Records/T249299
Vector WVUI IntegrationEdit
Short-Term Decision RecordEdit
What are your constraints?Edit
|Assumptions and requirements|
|The Web team needs to deploy the new search experience by early January 2021, so the solution to this problem can't take more than a few weeks to implement.|
|This decision doesn’t affect existing projects using webpack. Further, other projects do not need this decision in order to be unblocked.|
|Security audit & approval of Webpack is not feasible due to the way this software manages dependencies (see Appendix)|
|Using Webpack in Vector is controversial – both because of the security issues mentioned above, and because it would change the development process in a repository that is widely contributed to. This should be avoided in the short term, until an approach to build steps in skins and extensions is agreed on and implemented.|
|Using a build step in WVUI is not controversial. It stores its artifacts only upon release, and a separate location outside the repo, and allows for auditing through lib check-in commits.|
|WVUI can be treated as an internally-produced library that's external to MediaWiki (like OOUI)|
|Almost all of the Vue.js code for the new search experience lives in WVUI, with only a small amount in Vector. This means Webpack adds little value in the Vector repository|
A) All 3 release scripts are working and can be run in a local Docker container
B) Release process is automated in CI
What are your options?Edit
|Option 1: Do Nothing|
|Description||Don't use WVUI or Vue.js in Vector|
|Pros||We wouldn't have to make any major changes to Vector or its development process|
|Cons||We wouldn't be able to put the new search experience in Vector, other than by rewriting it without using Vue.js|
|Risks||The new search experience would not be deployed on time|
|Effort||A lot, the new search experience would have to be reimplemented|
|Costs||One-time duplication of work from reimplementing the new search; ongoing costs in continuing to argue about this, and other projects that want to use Vue facing similar issues.|
|Decision Type||Reversible: we could always integrate Vue.js and/or WVUI later|
|Option 2: Keep the proposed build step|
|Description||Use a Webpack build step in Vector, as currently proposed in the feat/search branch.|
|Pros||It's already implemented, so we wouldn't have to do any additional work|
|Cons||Introducing new Webpack build steps is highly controversial|
|Risks||Deployment of the new search experience would likely remain blocked because of the controversy, perhaps until a decision is reached on build step infrastructure|
|Effort||No effort, other than waiting for the medium-term build step decision to be made|
|Costs||The new search feature would be delayed until the medium-term build step exists, and is ready for prime-time production use on every wiki/page/user/device.|
|Reference||feat/search branch; see also webpack.config.js, App.vue and index.js in this patch (now abandoned)|
|Decision Type||Reversible: the build step could be removed and replaced with something else, with some effort|
|Option 3: Remove the Vector build step, use Vue and WVUI through ResourceLoader|
|Risks||This is a conservative solution by design and does not introduce new risks|
|Costs||(None, required patches already written or being written)
The only “cost” here for now is that Vue components that live directly in Vector (as opposed to in WVUI) should have manually-defined render functions instead of templates. If we have a ton of Vue components in vector that could become a hassle, but for a single App.vue component it’s trivial.
|Question||Who can answer?||Resolution, answer or action|
|Should we use WVUI as a small project to get comfortable with Rollup?||Give this a shot during the holiday code freeze|
|Is it worth shipping a compiler-free build of Vue 2 to be used in tandem with the new Vector search feature? This would have some performance advantages but would require manually writing render functions for any components embedded in Vector (the WVUI components would not be impacted). Example||Web & Perf teams||For now we should just proceed with the full compiler build of Vue, but this optimization could be explored in the near future for little effort if it is deemed necessary|
|Option||Option 3 (remove the Vector build step)|
|Rationale||Using Webpack in Vector is not necessary to complete the Search case study, and by doing so would introduce security vulnerabilities. ResourceLoader already exists as a tool to achieve a similar result, without any security vulnerabilities.This also removes any need to decide on whether/which 3rd party library to use for bundling Vector in the meantime, to meet our deadline.
All work in WVUI remains valid because the code lives in a separate library which can use its own build tools
|Data||See above Webpack section|
|Who||Roan & Web Team|
|Date||Now through January 2021|
|Informing||Vue Taskforce Stakeholders|
|Next Steps||Olga to create necessary phab tasks and assign them while adjusting OKRs based on new timeline|
So.. what is it about Webpack?Edit
Other local toolsEdit
Webpack is different. It executes hundreds of unaudited packages. Packages that even upstream Webpack won't know of or have reviewed (due to npm's indirect version resolution). Webpack's output file includes our own source code plus executable code generated by or embedded from those unaudited npm packages. Developers are then meant to attach this obscured file to their Git commit and deploy it blindly. All without having reviewed either the npm programs, or its output.
There are tons of npm packages written in a responsible manner suitable for production, such as Vue.js, Preact, Svelte, Rollup, etc. (Rollup is also used by upstream Vue.js for their own releases). These tend to have few or no external dependencies, and dependencies they do have are generally well-known, easy to audit, and have no further indirect dependencies.
Packages like this can be audited, and deployed via libs/, vendor/ or as their own repo/standalone service. See the Medium-term decision requirements for more details on what such a service could look like.
Packages like Webpack are quite different and consume the part of the npm ecosystem that's quite different in their principles. Packages that literally accept pull requests to remove one line of code in favour of an unfamiliar package with further dependencies, published shortly beforehand and carelessly updating them regularly to newer versions. Such "careless" updating isn't unheard of. After all, it's normal to install updates from a moderated repository such as Debian, Ubuntu, or your Mac/Windows operating system updates. These have a chain of trust that ends with a link between you and the distributor. There is no such link with NPM. NPM is effectively just a public FTP server, like GitHub. Anyone can submit anything, and there is no established practice or concept of trust in its operation. Hence, we must treat npm packages the same way we'd php-composer packages and JS libraries we use in production today. They require locally-owned redistribution and review of their source code. See the Medium-term decision requirements for more details on this.
Cross-team collaboration and productivityEdit
Beyond the above security concerns, using Webpack in the specific way proposed also harms productivity as every patch conflicts with every other patch. Developers would have to rebase a patch each time before being able to rerun its tests or perform code review, as well as to download the patch locally and re-generate build artefacts as part of every rebase (which we do in an isolated container, especially if you have prod access).
Webpack artefacts are currently monolithic and effectively binary, this means the workflow is incompatible with WMF backport and security deployment procedures. No patch can be cherry-picked. No patch can be reviewed in a private bug report, or transferred between production branches. Each subsequent week, requires re-patching every bug, first installing the insecure toolkit locally, and then committing this unreviewed binary to production without any tests. Last month, an XSS security issue was found in MobileFrontend. It proved rather difficult to patch, and actually motivated developers involved to disclose the vulnerability ahead of the general security release schedule, because carrying the security patch forward each week was impractical (T262213, Sam Reed, Security).