GitLab/Migration status

TL;DR On 2024-06-24 a decision to continue support for both Gerrit and GitLab as code forge solutions for the Wikimedia movement was announced on wikitech-l. Additionally we have decided not to revisit this decision for at least the next two years to reduce change fatigue across the teams responsible for supporting our code forges and related infrastructure. For quick answers to your questions about this decision, see the FAQ

Migration of projects to GitLab began in July 2022. Early adopters of GitLab sought a GitHub-like, self-service workflow. Developers who felt constrained by Gerrit jumped to GitLab. Since then, the growth rate has been steady.

Wikimedia GitLab projects over time.

There’s a demand for code hosting outside of Gerrit. Wikimedia Foundation-hosted GitLab has been a boon for these users—tool creators, engineers focused on data and analytics, and folks building services.

During this initial migration, we identified two distinct groups of repositories, each with their own needs:

1. Self-contained repositories – Repos without cross-repository dependencies that tend to have a small number of total contributors. These repositories were among the first projects to migrate to GitLab because their workflows fit well within GitLab. Examples include deployment tooling, Jupyter notebooks for analysis, and Toolforge tools.
2. Deeply connected repositories – These are repositories that combine into a single, working system. They require coordination across multiple repositories for developers to continue to be productive. Examples include Wikimedia Foundation-deployed MediaWiki extensions, skins, and core; as well as DNS and puppet—all are examples of groups of repositories which require cross-repository coordination at the individual merge level.

These two groups have different code review needs and practices that require different workflows. You can see the divergence of tool preference between these two groups in our developer satisfaction survey.

Satisfaction with Gerrit by functional role (left), Satisfaction with GitLab by functional role (right). from 2024 Developer Satisfaction Survey

We believe moving forward with the plan to migrate all repositories in Gerrit to GitLab will have an immediate and long-lasting negative impact on the productivity of developers working in these deeply connected repositories.

For this reason, we believe that we should stop migrating all repositories unconditionally, and instead keep our two systems: Gerrit and GitLab. Gerrit should remain for the use-case of deeply connected repositories. GitLab should remain for tools, analytics and machine learning, and services. The decisions about which specific projects will be in each system will be made by Release Engineering in consultation with the engineering or technical leadership for those projects.

MediaWiki and Operations repositories

As of early May 2024, there are 1,915 repositories remaining to migrate from Gerrit. Of these 77% are MediaWiki, extensions, and skins, or repositories within the operations namespace.

Our largest unmigrated repositories—repos in the MediaWiki and operations namespace—have our most complex contribution workflows. But GitLab lacks key features to support these workflows. GitLab’s missing features are necessary for the productivity of developers, deployment safety, and operational reliability. Tools we build on top of GitLab require investment and maintenance, and would necessarily be limited by GitLab itself and at risk of being broken by an upstream change.

Why not continue to migrate to GitLab?

The risks we’ve identified relate to missing features which sustain our workflows and our ability to deliver those features while working with GitLab upstream. Both missing features and the closed nature of GitLab upstream will impact our developer productivity.

These risks exist regardless of our choice to continue using GitLab Community Edition or moving to the GitLab Ultimate Edition (the highest tier of GitLab’s self-hosted paid offerings). Though the Ultimate edition carries additional risks (locked-in to a paid contract for our code) and provides one missing feature we’ve flagged (the absence of multiple reviewers), many of the core risks are the same between the two editions.

Missing features

We interviewed MediaWiki developers about their workflows and explored integrating MediaWiki workflows with GitLab. During this work, we flagged a list of missing features in both GitLab Community Edition and GitLab Ultimate Edition. The impact of these missing features is slower review and deployment cycles, especially for large features, and more human-caused errors in our production.

The key missing features we flagged are:

• Cross-repository dependent merge requests
  • In repositories with many contributors, it becomes necessary to coordinate merges to the mainline branch to avoid broken builds and to ensure we land features in the proper order.
  • GitLab Ultimate Edition supports this within a single repository via merge trains. But there is no support for cross-repository merge trains in any version of GitLab. No way to indicate that “this change in my extension depends on this change in core.”
  • We tried to mitigate this problem by introducing Zuul-CI into GitLab. But we found this difficult due to a long-standing issue with GitLab webhooks. We submitted code we believe resolves the issue; however, this is only the first part of a working solution.  
  • Impact: Without a way to coordinate merges cross-repository, we would see more CI failures and broken builds due to semantically incompatible patches landing on the mainline branch. We would also see more human-caused errors in production as there are no tool-level enforcements for ensuring all dependencies are met pre-merge.
• Stacked patchsets have several features we depend on in our deeply connected repositories:
  • Enforcing dependencies – Without stacked patchsets, we may merge broken code. A new feature within a single repository may depend on a small refactor. But the new feature may be able to merge without the refactor. Stacked patchsets enforce the merge order of patches within a single repository, ensuring changes merge as developers intend.
  • Discussing larger changes – Changes in core affecting many extensions are easier with stacked patchsets. A developer on one team may make changes to another team’s code stacked on top of a change in core. The other team may make a new patch, making a slight tweak on the original implementation. This is a granular means of discussing large changes that’s optimized for speed.
  • Forcing smaller changes – Small changes are core to our practice of trunk-based, continuous integration. Empirical software engineering research suggests and experience confirms: small changes are easier to review. In Gerrit, the unit of change is the commit, forcing us to create small changes. In GitLab, the size of a Merge Request depends on the person writing the code. More experienced developers may opt to make small Merge Requests, whereas newcomers may not.
  • Finding conflicts quickly – We resolve conflicts between code changes and our mainline branch at merge time. But because GitLab allows many commits to compose into one review (i.e., a Merge Request), this conflict may be large and difficult to resolve. In Gerrit, it’s typical for a large feature to span multiple reviews. When developing several features in parallel, large merge requests may require significant rework due to conflicts with other large merge requests—a state of permanent deadlock we avoid with stacked patchests.
  • GitLab has no support for stacked patchsets. We experimented with a tool called gerritlab to bring stacked patchsets to GitLab. GitLab upstream showed some interest in this tool and has since decided to experiment with their own version of it.
  • Impact: Without stacked patchsets, any medium- to large-sized feature would take longer to implement and merge as it requires more out-of-band, cross-team communication.
• Multiple reviewers – GitLab Community Edition lacked support for assigning reviewers when we evaluated it in 2020. GitLab has since developed a way to assign reviewers, but multiple reviewers are only available in its paid, enterprise offerings.
  • Impact: Without the ability to add multiple reviewers, changes will take longer to merge—reviewers go on vacation, may be unavailable on a certain day, or may be busy on a certain day. Meanwhile, other qualified reviewers are unaware of a change needing review.
• Dashboards, search, and visibility – Coordination requires visibility across a broader scope than a repo or a group. A simple example is showing all merge requests reviewed by a group of individuals across the whole instance. Another example is work needing my attention—this was rejected a year ago by GitLab, but exists in Gerrit as the user attention set.
  • Impact: Without the ability to find changes needing review, code review slows down.

GitLab upstream

While trying to find solutions to mitigate the impact of missing features, we worked with GitLab upstream. But we found many roadblocks along the way.

GitLab is a public company with an interest in paid licensing for the enterprise edition of its project. GitLab has little incentive to provide an easy way to customize or extend its software because there’s always a risk any extension would compete with one of its paid products. Their security update process also focuses on gitlab.com first, and self-hosted installations second.

While the GitLab Community Edition is licensed as open source, there is no meaningful way to extend it:

• There is no way to globally disable features of a self-hosted installation (issues, time-tracking, or snippets). We created a means of disabling features globally on a per-project basis via the API.
• There is no way to add any custom client-side code or even custom css.  
• The security release cycle is rapid and comes without warning—all installations learn about security vulnerabilities at the same time as attackers. We requested advanced notice of security releases, but this was rejected. This decision hinders our ability to maintain patches on top of our installation while keeping it secure.

The workflow of our deeply connected projects diverges from the core workflow GitLab supports. Without the ability to extend GitLab to support this workflow ourselves, and without the support of GitLab upstream, there is no way to add features to GitLab. For repositories that already fit into GitLab’s workflow, GitLab will continue to work well; for repositories that diverge from GitLab’s workflow, they will not work well.

Recommendation

For deeply connected repositories currently on Gerrit, we recommend that they remain on Gerrit and continue using the existing CI pipeline rather than continue to migrate to GitLab. This means that we will continue to maintain Gerrit and GitLab as separate code hosting and review systems—and continue to support and evolve the CI pipelines in each system.

This solution preserves the code review productivity of developers on some of our most production-critical projects, avoids reimplementing existing Gerrit features in GitLab, and preserves the pre-merge testing model integral to protecting production.

GitLab vs Gerrit, recommendations for repos

This subset of code must remain on Gerrit:

• MediaWiki core relies on features that GitLab lacks—the stacked-patch workflow and test dependency management. This reliance on Gerrit means MediaWiki core and those extensions and skins that track MediaWiki’s mainline code branch should remain on Gerrit, explicitly:
  • Wikimedia-deployed MediaWiki core, extensions, skins, and vendor
  • The subset of third-party extensions that track MediaWiki’s mainline
• SRE’s puppet code is likewise built around the Gerrit workflow. Specific SRE repositories that operate in lock-step with Puppet, explicitly:
  • Our DNS records repository.
  • Wikimedia Helm charts used to deploy services to our “Wikikube” Kubernetes.
  • The subset of code in the operations namespace on which Puppet has dependencies.

Beyond these repositories, the question of which code forge should host a repository becomes more ambiguous:

• Third-party MediaWiki extensions tracking release branches get little benefit from Gerrit’s features.
• PHP libraries used by code remaining on Gerrit may be developed on GitLab or Gerrit.
• SRE repositories deployed by Puppet but loosely coupled with the Puppet code can use either system.

Some repositories on GitLab may wish to move back to Gerrit. By already being on GitLab, these repositories have no explicit need for Gerrit's features; however, groups of developers may wish to keep all the code for which they are responsible within a single code forge to lessen the mental burden of working in two systems.

In the future, a repository initially developed on GitLab may become a repository that should use Gerrit; for example, a volunteer-developed extension that people wish to deploy to a wiki managed by the weekly train deployment. The extension developers may be happy on GitLab. Meanwhile, MediaWiki core developers may want to modify methods in core used by this extension.

Given the ambiguities, developing a resilient criteria set is only possible after discussions with code owners. Here are an initial set of recommendations for repositories:

• MediaWiki config, core, extensions, skins, and vendor MUST remain on Gerrit.
• Puppet code and repositories that are dependencies of Puppet code MUST remain on Gerrit.
• Submodules and libraries of the above SHOULD use Gerrit to limit developer overhead of two systems, but MAY opt to use GitLab.
• Services deployed to Wikimedia production SHOULD use GitLab to take advantage of the code deployment pipeline to production.
• Toolforge repositories SHOULD use GitLab to take advantage of the integrations between Toolforge and GitLab.
• For projects not in any of the groups above, developers SHOULD consult and make an agreement with the appropriate project’s steward.
• To lessen the overhead of switching systems, groups of developers MAY move repositories to either Gerrit or GitLab—this will be actively supported until the end of the calendar year and self-service / best effort support afterwards.

Uncertainty

• Added mental burden of two systems.
• What goes where? We need to refine the above criteria for what type of work lives in Gerrit vs. GitLab; this will be an ongoing process.
• Entrenching fragmentation – this continues the fragmentation of our ecosystem.
• We’re adding more points of failure and more surface for attack.
• During this multi-year migration, we’ve deprioritized our existing Gerrit and its Zuul-CI system install.
  • We need to prioritize upgrading Zuul to the latest version—this means deploying new technologies to be maintained.

Engineering effort & Long-term maintainability

• Gerrit upstream – Close coupling to Gerrit leaves us vulnerable to upstream changing course. This is mitigated by our close relationship with Gerrit’s upstream and our trust in their decision-making.

Maintaining two systems – We’ve been maintaining two systems during the migration; this decision entails a commitment from both Release Engineering and Collaboration Services to continue to do so.