Architecture Repository/Artifacts/Wikimedia technical systems: Architecture analysis

‎

Wikimedia Architecture Repository
Home | Artifacts | Process | Patterns

Wikimedia technical systems: Architecture analysis

An analysis of the state of Wikimedia's production wikis and the use of MediaWiki as the base software for over 900 instances with different requirements and needs

Last updated: 2023-06-09 by Novem Linguae
Author: Moriel Schottlender
Status: v1 published December 2022

Introduction

This document discusses the current state of the Wikimedia Foundation's production wikis and examines whether it is still a valid course of action to use a single monolithic piece of software for over 900 instances that have different requirements and needs.

This document is not intended to be a criticism of the work done by staff and volunteers over decades to support Wikimedia projects. It is meant as an analysis of the architectural risks of the current system, using examples to frame a discussion. This document does not argue for a specific implementation solution; it intends to start a discussion about the future direction of the system.

The Foundation's direction towards Knowledge as a Service requires us to shift not only how we think of new products, but the way we design and develop products in general. This document makes the case that there is a cost to developing features the way we do today – reactively and individually.

Our current infrastructure and architecture does not serve our plans for the future. While it does serve many different types of users, that very diversity has led to this impasse. It is extremely difficult to enhance or develop a part of the system further when the part itself serves incompatible purposes.

We need to rethink how we define our platforms, and how we define our architectural process. This document concludes with recommendations for next steps that can mitigate many of the problems that are described and set us up for succeeding in developing the strategic vision of the Foundation.

The state of MediaWiki's development

The Wikimedia Foundation runs the wikis – Wikipedia and 11 sister projects – on a cluster of servers that share the same base software: MediaWiki. Wikipedia's underlying code was originally built for collaborative editing of long-form text on the web. As more languages were added, so was the need for more features, and, eventually, other sites with similar missions. Communities formed around these sites to produce new types of content and shared knowledge. The software grew organically; with more and less overarching architectural guidance.

Today, it still serves as the backbone software to all of our wikis, including:

Project name	Instances (in production)	Edits per year (on all instances)	Views per year (on the biggest instance(s))
Wikipedia	~328	~196M	~120B (English)
Wikimedia Commons	1	~100M	~12B
Wiktionary	~180	~12M	~2B (English)
Wikibooks	~120	~540K	~118M (English)
Wikinews	~34	~4M	~97M (Russian) ~50M (English)
Wikiquote	~89	~470K	~95M (English)
Wikisource	~74	~5M	~210M (English)
Wikiversity	~17	~290K	~45M (English)
Wikivoyage	~25	~550K	~82M (English)
Meta-Wiki	1	~2M	~150M
Wikidata	1	~235M	~4B
Wikifunctions	0 (upcoming)	(upcoming)	(upcoming)
Total	~870

All of those multi-site and individual-instance projects – with all the differences and similarities between them – are all operated fundamentally using the same piece of MediaWiki software – but not all of them provide the same capabilities for reading and editing.

Over the years, many of those projects have received dedicated branching features that sat on top of the base MediaWiki software, while some features were included in the “core” of the software itself, allowing features to be shared between projects. But even so, all diverging features still operate within the context of the larger whole of our production systems, which means that, often, changing one behavior has unintended and undesired impacts on others.

Adding one feature to the system often requires accounting for many unrelated needs and competing features that are connected just because they exist as simultaneous requirements in the same software.

As a result, the software must do two things at the same time:

• It must support all required features in all projects and instances at the same time.
• It must allow for features to be extended when they diverge in needs between projects.

In practical development terms, as the projects grow, the added complexity of this requirement becomes much more impactful:

• Configuration becomes a mandatory complexity, to be able to adjust for the requirements of individual instances.
• Adding individual features or fixes for instances becomes much more difficult, since the underlying code must account for all instances.
• The system is less flexible since the pieces are so interconnected; disabling or removing features impacts unrelated projects or pieces of the software unexpectedly.

Historical attempts to use structure to manage code growth

Over the years, as the system expanded and grew organically, the stewards of the software – volunteers and staff – have made multiple attempts to manage the system architecture to allow for both sustained maintenance of features for the biggest sites (i.e., Wikipedia, or even just the English Wikipedia) and expanded behavior for the other projects and instances we maintain.

These attempts improved some of the tangled configuration and code that impacted new features, but still remained very much attached to the original idea of a single system that governs everything.

The challenges themselves and the way we attempted to solve them provides a glimpse into the problems of running over 900 instances with a single piece of software.

Separation of MediaWiki core and Extensions

Originally, the Wikimedia production software used one unified codebase, retrospectively called MediaWiki, and without the separation between “core” and “extension” functionality that exists today. To simplify management, we retrofitted the concept of Extensions to allow us to enable and disable modules of behavior and functionality in different scenarios, projects, or instances, and allow external third-party users to decide which behavior to utilize on wikis.

The question of “what should be included in core versus an Extension” was asked in individual contexts, and specific isolated decisions, without a cohesive long-term strategic plan.

Separating out Extensions

A good example of attempting to separate code into manageable pieces outside of the main MediaWiki system is the Cite Extension (and the citation capabilities in general). There are two functionalities related to citations: the footnote functionality (letting editors add citations to an article as footnotes) and the “cite this page” functionality (letting readers create a link to a version of an article, to be used externally). These functionalities were implemented separately but in the same Extension.

Having two conceptually different behaviors in the same Extension meant mixing concerns. It was not possible to install one without the other, and maintenance concerns were unnecessarily intertwined.

Eventually, this was corrected and separated out into two different Extensions that are self-contained and allow for easier maintenance. This separation had a minimal impact on third-party users: both Extensions continued to be bundled with MediaWiki for third-party downloads.

This illustrates the underlying problem in the attempts to manage the software: the separation and attempts to decouple the code is done reactively, as an attempt to improve configuration and operation of the multiple instances, based on a rough idea of what different parts of the monolith are meant to do and for whom they should be best arranged.

The Skin system

The Skin system is another example of this attempt to separate the system into extended modules.

The original Wikipedia used UseMod which provided a single frontend output that was, effectively, the “skin” of the site. A few years after the launch of Wikipedia, it was recognized that it is beneficial to separate the way the site looks into its own Extension, and the Skin system was created (and has been changed and updated since). This happened as part of the “phase3” rewrite that resulted in what we now call “MediaWiki”.

This has effectively separated the frontend "display" from the backend "content", but due to a lack of resourcing, the separation is not complete. Since the software was originally written to not have this separation, MediaWiki itself produces a significant amount of the output frontend HTML before the Skin system can intercept it, with areas varying on where responsibility lies. So, while the separation was done, it was done more as a means to make it easier to develop alternative skins adjusting most of the design and some of the functionality of the current system, rather than an architectural process to decouple the frontend from the business logic of the system.

This still produces some challenges for current development, specifically for mobile applications, and for the new efforts to modernize Wikipedia's web frontend.

The clash between Extensions and the concept of core

The problems from lacking a strategic architectural definition of core behavior, and the attempt to provide fixes that are reactive, are most evident in cases like the Notifications Extension that provides functionality for users to receive notifications about page changes and messages.

The Notifications Extension (named “Echo” internally) is the tool that controls sending and receiving notifications for logged-in users. It provides a structural interface to allow other features, including Extensions, to define their own notifications so that the user can be notified within the scope of new features. The Extension also allows users to dictate where they see notifications (email, web, and push notifications).

However, there are other, older sources of functionality for sending notifications; some notifications are created within MediaWiki core, implemented years before the Notifications Extension. This creates a situation where, to make a change to some notification, we must change the code in two or three different codebases, all related to the larger MediaWiki system.

This also exposes another challenge in the separation of concerns. Extensions are defined to be optional to the core MediaWiki software – meaning that the core cannot reference functionality that exists within an Extension directly. In the case of notifications, this exposes the problem where internal notifications and emails that originate from MediaWiki core cannot utilize the Notification Extension (because it is external to it) and therefore must define its own behaviors. This produces duplication of functionality, prevents MediaWiki core from using the more extendable engine behind the Notification Extension, and duplicates messages, translations, and outputs.

There was a plan to merge the functionalities together into a single system inside MediaWiki core (including the Watchlist notification system), but it was paused when the team was pivoted to other work.

This entangled behavior also produced challenges for various attempts to modernize our experiences, like providing push notifications or scheduled notifications, and to this day the email notifications from the different systems differ in the legal notices they provide recipients.

Diverging requirements between projects and communities

While we serve almost 900 instances in production (and more if counting beyond the mass-user production instances) many of those have requirements that differ substantially from one another. In many cases, the needed features and experiences diverge not only between the different projects – but often between different instances of the same project.

One of the most notable examples of this is the FlaggedRevisions Extension. There is a comprehensive overview of the different use cases between the wikis on Meta, and the extension has been known to be incredibly technologically complex, over-configurable, and consequently unstable.

Even within Wikipedia, different communities have utilized this feature differently:

• German Wikipedia, where the Extension was first deployed, uses the Extension to add informational flags about articles' quality.
  • Editors can flag articles as “sighted” – meaning they do not contain vandalism.
  • Reviewers can flag articles as “quality” – meaning the content was reviewed and judged valid.
  • The community treats this as the default process for reviewing changes to content. 99% of articles on German Wikipedia use this process.
• English Wikipedia then rejected the feature, and accepted it only after significant conceptual changes, as a trial. Currently, it is used to approve new edits by new editors in a process called “Pending Changes” as a way to restrict changes to pages without resorting to a full semi-protection of pages.

Between these two Wikipedia communities, the Extension's operation is used for exactly the reverse intent. In German Wikipedia, the assumption is of a starting quality, and the Extension is used to verify and enhance the details a reader may get for the quality of articles. In English Wikipedia, the process is meant to restrict publication from new, untrusted editors to reduce the vandalism readers may be exposed to.

The same Extension tries to support both behaviors, and has been used in about 50 instances for various purposes. It has been a continuing challenge to overcome its inherent complexity (without disturbing current usage) when adding a feature that relates to new editors and new revisions on wikis, as features must adjust to different types of behaviors of the same codebase on multiple wikis. This has become more and more impactful as the Foundation has prioritized improving the experience of new editors in the past years.

As a result, the instruction page on MediaWiki.org declares that the Extension is “clunky, complex and not recommended for production use” and has stated for many years that it “will no longer be installed on additional WMF wikis”. Some volunteer and ad hoc staff work over the past few years has been spent deleting large parts of the Extension's functionality in a fight back against the technical debt.

Our monolith increases the challenge of adding features

These are examples of features that were added – or were attempted to be added – into the existing system to support the needs of some of our communities, and have encountered significant development challenges.

These challenges manifested in significant delays in development, unforeseen necessary work beyond the original scope, features leaving behind a significant maintenance burden, or features being fully or partially abandoned with a decision to not ship the feature to some or all of the wikis.

Example 1: Change filters

Status	Completed. Deployed successfully.
Why did we work on it?	Investing in the critical workflow of the wiki communities, based on their demands and needs.
What did we do?	Extended the core change review mechanisms with filters, combining some existing features with novel review controls.
What went wrong?	This created a feature that is limited in scope and can't be easily extended.

This feature significantly improved and modernized the workflows for reviewing and approving edits by extending the interface with clear, approachable filtering tools. Although a significant departure from the original technical stack and only relevant to desktop and mobile-web users, we couldn't implement this outside of MediaWiki itself effectively. The extensibility provided to other tools, and the deep integration with the internals of MediaWiki, would not have been achievable in a separate service or system. Implementing this behavior outside of the core MediaWiki software would have resulted in the same challenges that currently plague the notification system.

Though the feature is widely liked and successful in its own terms, it is at the limit of where this functionality can go, and subsequent attempts to significantly add to it have risked production stability. When we add any new product into the Wikimedia ecosystem, we must also implement the ways that RecentChanges and Watchlist workflows can support and display its data.

Example 2: Adding partial blocks to Wikipedia

Status	Ongoing. Partially deployed. Paused.
Why did we work on it?	A highly requested feature by power users; directed by the Foundation's Board
What did we do?	The ability to block certain users from certain actions, rather than blocking users from the wiki altogether.
What went wrong?	Public commitment to the change without grasping the significant technical issues that the work would involve, without appetite for needed resourcing.

While attempting and working on adding features into the blocking mechanism, it became clear that the interconnected pieces were incredibly widespread. Interconnected code touched database operations directly as well as the user interface, and had not ever been modernized since the code was introduced 15 years previously. There were several now-invalid assumptions for which we had no plan as to how the area would develop in the future. After the first part of the feature was developed and tested, with other features still in development, the team were re-tasked, without any of the new plans being codified.

Fundamentally, the change meant that blocking needed to understand, for the first time, the concept of actions to be blocked. This fundamentally changed the contract and base assumptions of the original system around actions, which was previously a very loose conceptual area of the system.

Example 3: Wikipedia's new Frontend interface

Status	Ongoing. Partially deployed. Actively worked on.
Why did we work on it?	The current interface of Wikipedia is not compatible with the Internet's modern visual expectations.
What did we do?	Working on improvements to the visual paradigm of Wikipedia and its sister sites.
What went wrong?	Necessitated an increase in scope as changes to the HTML output of the page impact unrelated products. This has been particularly challenging when code ownership within the monolith is not well defined.

The current work to update Wikimedia projects' interface has been ongoing and is producing new visual paradigms for our projects. This work updates the “Vector” skin, the visual interface skin that is the default look of our projects, by creating an alternative “Vector 2022” skin and working on incremental changes.

There are a few technical challenges that the team has encountered that made this work (and several similar attempts in the past) require a much larger scope, longer working time, and bigger technical investment. These are mostly evident because MediaWiki has no clear separation between interface and business logic:

• Some of the work must go back into MediaWiki itself and change the base operation.
• Changes in visual paradigms mean adjusting unrelated pieces of software. An example of this is having to change and technically refactor some of the Notifications Extension to allow for the different visual approach of Vector 2022.

A related problem to the above that added complexity for the team is that code ownership within the monolith is poorly defined, which means the team must work outside their scope to either find others who can help with the problems that appear in unrelated code, or they must do so themselves.

Another challenge that is produced by the monolith is the assumption that all wikis can use the same paradigm of display. It is unclear if Vector 2022 can work on Wikidata, for example, without diverging technical efforts.

This new design is deployed as an opt-in trial to all Wikimedia wikis, and is shown by default on several Wikipedias, even as development is ongoing. It has been successful overall in changing our design paradigm, but the technical hardships that the team encounters have extended the work beyond its original technological scope.

This also means that there are several design improvements that the team needed to either re-scope or give up on, since implementing them in the current system would require a much larger internal investment to make the underlying system compliant.

Example 4: The Translate Extension

Status	Critical system. Deployed. Known conceptual issues.
Why did we work on it?	In addition to translating MediaWiki interface messages, there was a need to create a user-friendly way to create translations for long-form segments of text on our projects.
What did we do?	We created an Extension that provides in-place replacement based on the requested language code to long-form segments inside a page.
What went wrong?	The behavior answers a very specific use case, but is used for many other use cases in a way that complicates the rendering and parsing of pages.

The Translate Extension was originally meant for translating user interface messages that were a part of the MediaWiki software. A later change added a way for editors to produce translatable pages of long-form text that can be translated on wiki. Readers can then read these pages in the language of their choice.

However, since its inception, users have created and built writing and translation workflows on top of the Extension, and many of those are now producing challenges to maintain as well as to add more translation features in the overall system.

The Translate Extension is widely used and is a crucial part of editors' workflow. As such, it manages a large amount of content written by users. This means that changes to the behavior of the Extension need to be done very carefully to avoid breaking user-written content, which makes architectural changes challenging to design and implement. As a result, we often end up adjusting other products and features to fit with the current behavior so as not to risk impacting users.

In the past few years, a lot of work has gone into normalizing the code, fixing parsing challenges, and working towards compatibility with future features.

MediaWiki's inherent assumptions about content and structure

At its core, MediaWiki expects its knowledge items to be free-form wikitext pages. This means that the underlying system makes these base assumptions:

• How to calculate the dependencies of changes – when to invoke cache invalidation, and to what pieces of the content, when a change is made
• How it should display the history of pages
• How it calculates and presents diffs between changes
• How changes are presented and displayed in the RecentChanges and Watchlist features
• What editing methodology (full-page wikitext) it expects
• How to store the content and its relationships, mostly in relational database tables

All of those assumptions inherently assume a full page as the structure and expect changes to conform to how MediaWiki core expects a page to behave. It's possible to add new content types that are not pages, and MediaWiki's core behavior allows for those. However, when adding new forms of content, there's a need to expand the scope to make sure that the implementation teaches MediaWiki about how to change its assumptions about the new content type.

In technological terms, it means adding a feature that relies on new content types often involves also implementing:

• How it would be stored
• How diffing is calculated, and how the history page is displayed
• A way for RecentChanges and Watchlist to intercept edit events
• Dependency tracking and cache invalidation for changes
• Integration with other Extensions (or core behaviors) – like defining how notifications should work for the specific workflow

In practical terms, this means that when we implement a different type of content paradigm, we find ourselves having to re-implement the basic behaviors of how to present the diffing, history and changes to override MediaWiki's base expectations of linear history of a full text page.

Example: Wikidata

Wikidata has encountered this problem most significantly, seeing as its own internals are completely redefining what content means: small structured objects with hierarchical and relational connections, rather than a full page. As a result, Wikidata not only needed to redefine the behavior of the core system, it is still encountering systemic issues relating to its scalability – in storing inside the relational database, and most notably, in tracking the dependencies that allow for changes in a Wikidata item to invalidate the cache of Wikipedia pages where those items are used.

These challenges are ongoing, and there are multiple attempts to improve them within the confines of the current system and its inherent assumptions.

Example: Section editing

Attempting to add section editing – the ability to only edit one section at a time rather than load the entire article – is another challenge that emerges from the base assumptions of MediaWiki. While this is possible in the raw wikitext editor from a technology standpoint, it's incredibly challenging to develop user-friendly utilities, like citations.

Citations are defined throughout the article, and can be reused throughout the text. If a citation was defined in a section, it can be reused in a followup section by reference without including all of its details. This means that if we only load the followup section – where the citation is mentioned but lacks details – the editor and system cannot know what the citation details are. We must load the entire page content and parse it in order to display the full information for a single section.

This is also true for the numbering of citations; the numbering is done during parsing, which means that there is no way for the system to know what number to give an individual citation inside a detached section without knowing how many distinct citations came before it in the overall page.

Challenges from the user's perspective: The projects have different expectations

While all projects are inherently built on MediaWiki and share the general mission statement of the Foundation and movement, many of these projects and communities have different expectations of behavior from their respective products. More impactfully, many of the wikis try to build types of knowledge that are at odds with the same base assumptions that MediaWiki holds, its structure and the way it should be edited.

There are multiple ways to examine the Wikimedia projects and consider the features and capabilities that unite them versus the ones that may be unique to certain projects.

Defining units of knowledge

While all our projects provide collaboratively curated knowledge to the world, some projects define different units of the knowledge.

The most obvious is Wikipedia, where the knowledge unit is a full-page article. This seems to also be true for Wikivoyage and Wikinews. There's an argument to be made about potentially being able to split an article into sections that can also be valid knowledge units on their own, barring the technical challenges explained in the sections above. But in general, even those depend on the narrative context of the article, and the community writes and edits the knowledge on Wikipedia intended for a full-length encyclopedic article.

In contrast, Wikiquote's units of knowledge are individual quotes that can stand on their own and are attributed to the speaker. They are currently edited within a single, long-form page as a list, but that seems to be because that is the way editing works by default, rather than specifically what Wikiquote needs.

Projects like Wiktionary and Wikispecies are a bit more elaborate in their definition of the units of knowledge and, arguably, contain multiple types of units of knowledge within them. Whatever the units of knowledge are, the fact that these projects have the same content structure on every page (currently reliant on social agreement and templates) indicates that the content has a more granular structure than a full-page article. If that structure can be exposed and edited in a more concise manner, it would help expose that type of content to the world, and make it more readable, more accessible, and more reusable.

Wikimedia Commons has a clearer definition of the unit of knowledge – a media item – and for the most part, it attempts to provide this to both editors and consumers. Over the past few years, there were changes made to add structured content and metadata to those items. However, the storage through MediaWiki itself has not been ideal. MediaWiki still treats images as pages and the image itself as an embedded piece inside it. While many properties are now structured (and are editable separately) the sections for the copyright and some other details are still edited with the long-form wikitext editor. That, too, creates some challenges when fetching images and media, especially due to some lack of consistency in the way those details are shaped.

Wikidata's units of knowledge are units of structured data that relate to one another, can be fetched directly and translated at all levels. This is the extreme opposite of the full-page free-form text content of Wikipedia, despite the fact that it uses the same system – and relational database storage mechanism – as the base software. As a result, the software had to be extensively adjusted (with the “Wikibase” Extension) and all features on Wikidata are fairly standalone and adjusted to its unique operation and storage concerns.

Editing experience

The core concept of MediaWiki is one in which a single block of wikitext markup is rendered into a large block of prose that takes up a web page on its own. This fits projects that are built on more of a narrative-like page structure, like Wikipedia, Wikivoyage and Wikinews. It does not fit as well with products that require some sort of structured content.

Wikidata is an example of a wiki that relies completely on structured data, and has almost completely moved away from the paradigm of a full-page text editor. While the paradigm of “everyone can edit” remains, and the ideas of viewing history, diffs, discussions and collaborative edits are emphasized in the process, the actual editing process is radically different than the one given by default with MediaWiki. That meant changing not only the editor itself, but the way we enable editing of the relationships between objects and how we store them.

Wikisource has two quite different editor experiences: the initial transcription of the page is done by giving the user a side-by-side view of the image of an original scanned book page alongside a free-text editor, to verify and correct the OCR automatic transcription. The follow-up actions, where editors add markup, styling and formatting is similar to the Wikipedia text editors. Unlike Wikipedia, however, Wikisource editors work on mixtures of single book pages and combined book texts to edit the full work.

Semi-structured projects like Wikiquote, Wiktionary and Wikispecies are currently using the full-page wikitext and visual editors, but arguably, the structured nature of their content suggests that this doesn't completely answer their need. The fact that the communities in these three projects take great pains to set up rules about how to structure the content within the wikitext editor – usually using inline templates and many guidelines the editors must learn – indicates that the editing capability of these projects may not hold the same experience expectations as the one that's provided. (This need is also apparent in other areas, such as Wikivoyage's "Listings" tool, which is slowly being removed because of these complexities.)

Moreover, adding editing functionality to these wikis to support an attempt at a structured content meant overloading current features (like templates and gadgets) to override default behavior and try to make the underlying MediaWiki editing assumptions behave as the users need it to.

Collation and groupings

MediaWiki primarily uses “categories” to collate and group items together. Functionally, MediaWiki categories are more similar to tags; they can be circular, non-hierarchical, inherited from other pieces of content, and pieces of content can contain many of them.

There are significant problems with the way MediaWiki defines and stores categories and their relationship to content that produce serious problems for extending its base features. An example is how categories are fetched, which stems from the limitations of storing them in tables in a relational database. Ideally, content would use generalized categories, and users would be able to fetch multiple intersections, for example using the combination of “women”, “european”, and “physics” to fetch a list of European women physicists. But this type of query is usually expensive because relational databases are not optimized for it.

As a result, our communities overloaded categories to be very specific and hierarchical, like “French women scientists” which means that they must always balance needing to define specific categories and generalized ones.

While Wikipedia and many other projects utilize the categories system, we have multiple projects that collate their content using vastly different methods.

Wikivoyage, for example, has a standalone Extension that creates breadcrumbs based on a recursive examination of location relationships, and uses that to collate articles based on location. That behavior and Extension don't work well with our caching systems.

Wikiquote collates its content by pages – where each page has a list of individual quotes from a single author. As discussed above, this seems to be done only because of the shape of the underlying system rather than because of the actual need, and the full-page article is used more as a categorization and collation method for the individual quote items.

Wikisource uses subpages to collate pages of books and publications; books are generally in the main namespace, with individual chapters as sub-pages. Categories are then used to collate different books more broadly by topic and subject.

Wikimedia Commons uses a combination of categories and manually curated editorial collection pages to group and collate media objects around themes and topics. It is possible that groupings will start to coalesce around the structured data that is being added, but the system will need to adjust itself for this behavior to allow the units of knowledge to be fetched based on the properties, and consideration will need to be given to where that structured data is stored within the database.

Conclusion

There are many challenges that the movement faces as we move towards the modernization of our products and technologies. Many of those challenges impact the way we've been thinking about our products in light of the changing world – be it changing assumptions about reading, expectations about media, or the ability to share our information with knowledge base systems as a form of Knowledge as a Service.

While Wikipedia serves its communities and is the top-referenced knowledge base online, it has also reached a developmental plateau. More specifically, by fitting all features to work within the assumptions and current workings of MediaWiki, we are increasing the amount of features that have run into challenges that block further development.

Recommended next steps

As we move towards our future, we need to re-examine our assumptions about our own projects, our way of work, and the way we design products. We have to stop letting our codebase and current production tools dictate how our products are made.

We must start looking at our projects as an entire system that needs to support the strategic vision and allow us to develop new products in a sustainable and scalable way. This change of paradigm means changing the way we look at our development process as well as the way we define and execute new features for users.

Finding the core shared capabilities

While this document discusses how our wikis diverge from one another in key terms that make the use of an underlying single monolithic architecture more challenging, there's no doubt that there is some core behavior that is shared – or should be shared – between all of our projects.

The idea that we should look at the divergence and ask ourselves whether the architecture should be different and allow for more differences between projects does not mean we need to completely develop distinct architectures per project.

But the question of what is the core behavior has not been systematically answered, and can lead to several revelations about how we view the mission of our projects, and what audiences and experiences we cater to.

We need to make the question of what is core behavior – what is the essence of our products and mission that we provide our users – a central question, rather than a reactive technological one.

As we explore shared capabilities, we reveal the potential for a true technological vision for our systems that can lead us to a sustainable, scalable architecture.

Example: Authentication and authorization

As an example, authentication and authorization is not only a shared capability for our projects, it requires the sharing of information between projects to allow users to participate in multiple projects under the same account. Currently, much of that behavior is inside MediaWiki in a way that forces a lot of external tools to use MediaWiki itself as the validator of actions, which then produces many problems related to security, trust and safety, and extendibility of the system in general.

There is a current initiative to model and explore a decoupled authentication and authorization system that can work with MediaWiki and the other services and projects we are running.

Invest in developing tools and platforms that support core capabilities

One of the reasons that the current and historical state of the system repeats itself is because new products do not have a proper technical solution to choose from when designing their products. As a result, most solutions utilize the current system, whether it is fully meant for it or not.

We need to make sure we expand our technological capabilities to allow for products to consider what is the best way to deliver their experiences rather than limiting new capabilities by forcing them into the current system.

Example: Structured data

As an example, both Wikidata and Wikimedia Commons' structured data initiatives intend to establish relationships between objects and improve discoverability. However, both of these products are hosted within the paradigm of MediaWiki – a software that initially was built as a full-page collaborative editing system that stores and tracks its data within relational database tables. We created some supporting elements around these products to allow us to fetch and filter objects based on their relationships directly, but these products are still inherently limited because the structure they're built on top of is not, inherently, meant for this operation.

While we've added support for structured data through both of these initiatives, they are also both limited by what they can let us do with that data structure.

Without developing and architecting a platform that supports the goal of these structured objects, it will be impossible to fully utilize the power of these features, expand on them, and provide more tools and features that support the vision, like graphQL, machine learning algorithms, and others.

Codifying architecture agreements

In order to stop being reactive and mitigate the problems this document explains (and more), we have to formulate a product architecture vision, process and principles and codify them for all to follow. We need a description of the desired system and a way to codify our working agreements on how to develop new products and how to support new technologies. There needs to be a commitment to follow these principles and processes, and accountability to make sure we stay within that commitment as we continue to work on the system.

The challenges we face are not just technical, they are also social – a sociotechnical environment that has the social interactions and the technology woven together tightly. This is true for both our user interactions as well as the way we work within the Foundation and on our codebase and products. We need to start working with an architecture that is not just “how code is structured”, but rather a codified set of agreements and processes to transform the strategy into working technology, where platforms are developed to answer the ongoing needs.

To make sure that we proceed and develop a system that allows us to deliver the types of new products we are aiming for, we have to understand the relationship between the social aspects – how we develop, what is the strategy we follow, who are our stakeholders, etc – and the technology we need to create to support this vision.

Example: Decision making

One of the biggest challenges that the Wikimedia movement has had throughout the years is figuring out the process of decision making, especially in the technical realm. Many iterations of decision making processes were attempted, and there is a process today that is actively being iterated on in the shape of the Technical Decision Forum.

However, the shape of our system makes all decisions complex by definition. Changes to the system can impact a large amount of other unrelated features and properties, which means that decisions about technical implementations and technical directions often must involve many stakeholders, sometimes ones who are not necessarily directly involved with the original request.

In contrast, a system that has well codified architectural agreements with well established boundaries would reduce the number of decisions that require a heavy, encompassing consultation as part of the decision making process. This would make our decision process easier, much more flexible and iterative, and would mean teams' work can be more accurately predicted within their scope.