GitLab/Initialization/Technical Design Document
Summary
edit| 1 | Project Name | GitLab: Ansible Installation and Performance tests |
| 2 | URL |
|
| 3 | Audience |
|
| 4 | Countries targeted |
|
| 5 | Language(s)
(ISO 639-1) |
|
| 6 | Development Type |
|
| 7 | Ticket system |
|
| 8 | Delivery timeline and status | Initial Collaboration
Testing Setup Building Ansible playbook for GitLab Server Building Ansible playbook for GitLab Runner Performance Testing MVP Delivering |
Introduction
editThe GitLab Initialization project aims to provide WikiMedia Foundation’s (WMF) Release Engineering team (RelEng) with a minimum viable product (MVP) type GitLab installation.
GitLab is a version control and Continuous Integration (CI) system. It is a web application written in Ruby on Rails accessing Postgresql and Redis as databases and other shared storage to store the source code repositories. There is an Enterprise Edition (EE) which is licensed software and a Community Edition (CE) which is open source and free edition. We will be using GitLab CE.
Currently WMF uses a combination of tools including gerrit/zuul/jenkins in a similar function. The usage is about 12 gitops per sec, 1300 repositories, 800 GB size, 1000s users, 250 active/month, about 30 CI VMs with 450,000 CI minutes/month, so 43800 min/month, approximately 11 servers active at any given point in time. CI VMs run under WikiMedia Cloud Services (WMCS).
The current gerrit/zuul/jenkins system has a limited high availability setup with warm standbys in an alternate data center that could be activated within hours. The CI VMs however run on WMCS which is only available in the eqiad data center. We have not tested a failover of the systems.
Under the MVP approach RelEng plans to transfer a number of WMF repositories, starting with a subset of RelEng's repositories and working with volunteer WMF teams.
The MVP installation is oriented towards minimal change from the current setup in terms of infrastructure. In terms of function the MVP setup is affording us the ability to gain experience with being good project owners of a GitLab install: maintenance tasks that must be automated, production systems integrations, ensure we can run a testing environment that matches a production environment, explore and establish login and authentication options, and establish deployment and upgrade cadence. These are all part of the setup phase (https://www.mediawiki.org/wiki/GitLab/Roadmap#2._Setup) of the GitLab roadmap.
A team of consultants from Speed&Function has been contracted to perform the MVP installation. See T274458 as the main entry point in phabricator.
The Service Operations team in Site Reliability Engineering (SRE) is hiring 2 SREs that will be responsible for the GitLab infrastructure. These SREs will be part of a FY 21/22 project to review the GitLab installation and reinstall GitLab according to the review. The team of consultants is tasked with providing documentation necessary for the review and reinstallation.
Installation parameters
editNetwork position
editGitLab MVP is installed in the production network due to access and data storage requirements. To avoid a dependency between the traffic infrastructure and GitLab MVP it is not behind the traffic infrastructure but has its own external IP addresses, similar to gerrit, a machine address and a service address. A test installation of GitLab is running in WMCS. (T278197)
Compute Infrastructure
editGitLab MVP runs on a single standalone server in the production network running a Debian based distribution. Performance tests have shown that a single server is capable of handling WMF’s traffic and processing needs. The server is managed with Puppet: firewall, certificates, OS packages etc. The GitLab MVP application is managed by Ansible for installation and configuration. High-availability considerations will be addressed in the upcoming GitLab reinstallation project.
Storage Infrastructure
editGitLab MVP uses internal storage for all purposes.
GitLab Application Installation
editGitLab is installed through GitLab provided omnibus packages and configured through the gitlab.rb file. HTTPS is implemented through the included nginx server, while certificates are managed through WMF puppet. GitLab git via ssh access is implemented through a secondary SSH server on port 22 on the service IP address. CI/CD integration runs on GitLab runners in WMCS. GitLab runner location considerations will be addressed in the upcoming GitLab reinstallation project.
Authentication
editGitLab MVP will use WMF CAS-SSO Apereo Single Signon. Group integration is not available in GitLab CE and will be done manually and automated in a later effort by WMF.
Monitoring
editThe GitLab MVP host is monitored via Prometheus and Icinga for OS infrastructure parameters. Dashboards for OS and GitLab are available in Grafana. Basic uptime checks are in Icinga.
Logging
editGitLab MVP logs everything to /var/log/gitlab/ directory. Files are mirrored to the foundation’s logstash server.
Backup
editGitLab MVP uses GitLab built-in backup routines to generate backups via cron. There is daily full backup of everything but build artifacts and logs. Backups are stored locally. WMF bacula picks up backup files once a day.
Security and Updates
editOS security is the responsibility of the SRE team.
GitLab security is the responsibility of the Release Engineering team.
Architecture of the Performance tests
editThe GitLab Performance Tool (GPT) is built and maintained by the GitLab Quality Enablement team to provide performance testing of any GitLab instance. The tool has been built upon the open source tool K6.io and provides numerous tests that are designed to effectively test gitlab.
The tool can be used both manually and automatically, with us doing the latter for automated testing of reference environments via Pipelines.
GPT is used by the GitLab Quality Enablement team to continuously perform test GitLab on environments based on our Reference Architectures that have been built with the GitLab Environment Toolkit. For more information please refer to our blog post - How our QA team leverages GitLab’s performance testing tool (and you can too).
The purpose of the performance test is to see whether the configured VM instance for GitLab is capable of delivering the transaction rate specified, i.e. about 12 gitops per sec on 1300 repositories with a 800 GB size. Thousands of users with about 250 active/month.
The performance of a VM with 8 processors and 16 GB of memory was adequate in our tests on WMCS. Tests in production on a VM with 8 cores and 12 GB are pending.
Test Scenarios and outputs
edit- Git
- API
- WEB
- Output
See Appendices
GitLab automated configuration
editAnsible playbook
editAnsible is used to set up and configure Gitlab Server and Gitlab Runner. Ansible was selected to avoid conflict with foundation’s puppet configuration management system, while still documenting and automating the setup. The following components are installed via Ansible:
- The gitlab application and its configuration
- The secondary SSH server to be used for git
- The gitlab runner responsible for CI/CD jobs
GitLab CI is a powerful tool that can be used for a number of things, including infrastructure as code, deployments and GitOps.
Before running Ansible playbook it needs checking the ability of ssh-ing the target hosts and have sudo permissions there
Automated GitLab Server installation
edit- GitLab (as a platform) is installed and configured with provided Ansible playbook which can be found on WMF's Gerrit: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible/
- Secondary SSH server for git
- Hosts to deploy GitLab Servers need to be configured in [gitlab_servers] section of 'hosts' file (it's an Ansible inventory configuration file)
- Ansible playbook parameters are set up on host-by-host basis in host_vars/<hostname> YAML files
- List of available playbook parameters can be found in gitlab_server/defaults/main.yml file
- To run Ansible playbook just execute install-gitlab-server.sh script
Automated GitLab Runner installation
edit- GitLab Runner is installed and configured with provided Ansible playbook which can be found on WMF's Gerrit: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible/
- Hosts to deploy GitLab Runners need to be configured in [gitlab_runners] section of 'hosts' file (it's an Ansible inventory configuration file)
- Ansible playbook parameters are set up on host-by-host basis in host_vars/<hostname> YAML files
- List of available playbook parameters can be found in gitlab_runner/defaults/main.yml file
- To run Ansible playbook, execute install-gitlab-runner.sh script
Appendices
editA: Decision Log
editB: Tests: Scenarios
edit| NN | Name | Description | Endpoint |
| 1 | api_list_group_variables | List group variables
Setup stage: Create group and multiple group variables Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/group_level_variables.html#list-group-variables |
GET /groups/:id/variables |
| 2 | api_list_project_variables | List project variables
Setup stage: Create group, project and multiple project variables Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/project_level_variables.html#list-project-variables |
GET /projects/:id/variables |
| 3 | api_new_branches | Create a new branch in the repository
Setup stage: Create group and project Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/branches.html#create-repository-branch |
POST /projects/:id/repository/branches |
| 4 | api_new_commits | Create a commit with multiple files and actions
Setup stage: Create group and project Teardown stage: Delete group Dicumentation: https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions |
POST /projects/:id/repository/commits |
| 5 | api_new_group_variables | Creates a new group variable
Setup stage: Create group Teardown stage: Delete group Documentation https://docs.gitlab.com/ee/api/group_level_variables.html#create-variable: |
POST /groups/:id/variables |
| 6 | api_new_issues | Creates a new project issue
Setup stage: Create group and project Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/issues.html#new-issue |
POST /projects/:id/issues |
| 7 | api_new_project_variables | Creates a new project variable
Setup stage: Create group and project Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/project_level_variables.html#create-variable |
POST /projects/:id/variables |
Tests: Git
edit| NN | Name | Description | Endpoint |
| 1 | ls_remote | Git Refs List via HTTPS
Controllers: Repositories::GitHttpController#info_refs |
GET /:group/:project.git/info/refs?service=git-upload-pack |
| 2 | pull | Git Pull via HTTPS to pull from master having another branch locally.
Documentation: https://gitlab.com/gitlab-org/quality/performance/-/blob/master/docs/test_docs/git_pull.md |
GET /:group/:project.git/info/refs?service=git-upload-pack
POST /:group/:project.git/git-upload-pack |
| 3 | push | Git push commit(s) via HTTPS.
Documentation: https://gitlab.com/gitlab-org/quality/performance/-/blob/master/docs/test_docs/git_push.md |
GET /:group/:project.git/info/refs?service=git-receive-pack
POST /:group/:project.git/git-receive-pack |
| 4 | pull_bulk | Git BULK Pull via HTTPS to pull from master having another branch locally.
Documentation: https://gitlab.com/gitlab-org/q uality/performance/-/blob/master/docs/test_docs/git_pull.m |
GET /:group/:project.git/info/refs?service=git-upload-pack
POST /:group/:project.git/git-upload-pack |
Tests: API
edit| NN | Name | Description | Endpoint |
| 1 | groups | List groups.
|
GET /groups |
| 2 | groups_group | Get all details of a group.
Documentation: https://docs.gitlab.com/ee/api/groups.html#details-of-a-group |
GET /groups/:id |
| 3 | groups_group_subgroups | List a group’s subgroups.
Documentation: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-subgroups |
GET /groups/:id/subgroups |
| 4 | groups_issues | List groups issues.
Documentation: https://docs.gitlab.com/ee/api/issues.html#list-group-issues |
GET /groups/:id/issues |
| 5 | groups_merge_requests | List groups merge requests.
Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#list-group-merge-requests |
GET /groups/:id/merge_requests |
| 6 | groups_projects | Get a list of projects in this group.
Documentation: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-projects |
GET /groups/:id/projects |
| 7 | projects | Get a list of all projects.
Documentation: https://docs.gitlab.com/ee/api/projects.html#list-all-projects |
GET /projects?order_by=id&sort=asc
GET /projects?pagination=keyset&order_by=id&sort=asc |
| 8 | projects_deploy_keys | Get a list of a project’s deploy keys.
Documentation: https://docs.gitlab.com/ee/api/deploy_keys.html#list-project-deploy-keys |
GET /projects/:id/deploy_keys |
| 9 | projects_issues | List project issues.
Documentation: https://docs.gitlab.com/ee/api/issues.html#list-project-issues |
GET /projects/:id/issues |
| 10 | projects_issues_issue | Get a single project issue.
|
GET /projects/:id/issues/:issue_iid |
| 11 | projects_languages | Get languages used in a project with percentage value.
Documentation: https://docs.gitlab.com/ee/api/projects.html#languages |
GET /projects/:id/languages |
| 12 | projects_merge_requests | Get all merge requests for this project.
Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#list-project-merge-requests |
GET /projects/:id/merge_requests |
| 13 | projects_merge_requests_merge_request | Get information about a single merge request.
Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr |
GET /projects/:id/merge_requests/:merge_request_iid |
| 14 | projects_merge_requests_merge_request_changes | Get single MR changes.
Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr-changes |
GET /projects/:id/merge_requests/:merge_request_iid/changes |
| 15 | projects_merge_requests_merge_request_commits | Get a list of merge request commits.
Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr-commits |
GET /projects/:id/merge_requests/:merge_request_iid/commits |
| 16 | projects_merge_requests_merge_request_discussions | Gets a list of all discussion items for a single merge request.
Documentation: https://docs.gitlab.com/ee/api/discussions.html#list-project-merge-request-discussion-items |
GET /projects/:id/merge_requests/:merge_request_iid/discussions |
| 17 | projects_project | Get single project.
Documentation: https://docs.gitlab.com/ee/api/projects.html#get-single-project |
GET /projects/:id |
| 18 | projects_project_pipelines | List project pipelines.
Documentation: https://docs.gitlab.com/ee/api/pipelines.html#list-project-pipelines |
GET /projects/:id/pipelines |
| 19 | projects_project_pipelines_pipeline | Get a single pipeline.
Documentation: https://docs.gitlab.com/ee/api/pipelines.html#get-a-single-pipeline |
GET /projects/:id/pipelines/:pipeline_id |
| 20 | projects_project_services | List project services.
Documentation: https://docs.gitlab.com/ee/api/services.html#list-all-available-services |
GET /projects/:id/services |
| 21 | projects_releases | Get project releases.
Documentation: https://docs.gitlab.com/ee/api/releases/#list-releases |
GET /projects/:id/releases |
| 22 | projects_repository_branches | Get a list of repository branches from a project, sorted by name alphabetically.
Documentation: https://docs.gitlab.com/ee/api/branches.html#list-repository-branches |
GET /projects/:id/repository/branches |
| 23 | projects_repository_branches_branch | Get a single project repository branch.
Documentation: https://docs.gitlab.com/ee/api/branches.html#get-single-repository-branch |
GET /projects/:id/repository/branches/:branch |
| 24 | projects_repository_commits | Get a list of repository commits in a project.
Documentation: https://docs.gitlab.com/ee/api/commits.html#list-repository-commits |
GET /projects/:id/repository/commits |
| 25 | projects_repository_commits_commit | Get a specific commit identified by the commit hash.
Documentation: https://docs.gitlab.com/ee/api/commits.html#get-a-single-commit |
GET /projects/:id/repository/commits/:sha |
| 26 | projects_repository_commits_commit_diff | Get the diff of a commit in a project.
Documentation: https://docs.gitlab.com/ee/api/commits.html#get-the-diff-of-a-commit |
GET /projects/:id/repository/commits/:sha/diff |
| 27 | projects_repository_compare_commits | Compare commits.
Documentation: https://docs.gitlab.com/ee/api/repositories.html#compare-branches-tags-or-commits |
GET /projects/:id/repository/compare?from=commit_sha1&to=commit_sha2 |
| 28 | projects_repository_files_file | Get information about file in repository.
Documentation: https://docs.gitlab.com/ee/api/repository_files.html#get-file-from-repository |
GET /projects/:id/repository/files/:file_path |
| 29 | projects_repository_files_file_blame | Get blame information about file in repository.
Documentation: https://docs.gitlab.com/ee/api/repository_files.html#get-file-blame-from-repository |
GET /projects/:id/repository/files/:file_path/blame |
| 30 | projects_repository_files_file_raw | Get raw file from repository.
Documentation: https://docs.gitlab.com/ee/api/repository_files.html#get-raw-file-from-repository |
GET /projects/:id/repository/files/:file_path/raw?ref=master |
| 31 | projects_repository_tags | Get a list of repository tags in a project.
Documentation: https://docs.gitlab.com/ee/api/tags.html#list-project-repository-tags |
GET /projects/:id/repository/tags |
| 32 | projects_repository_tree | Get a list of repository files and directories in a project.
Documentation: https://docs.gitlab.com/ee/api/repositories.html#list-repository-tree |
GET /projects/:id/repository/tree |
| 33 | search_global
(only EE) |
Global Search API.
Documentation: https://docs.gitlab.com/ee/api/search.html#global-search-api |
GET /search?scope=* |
| 34 | search_groups
(only EE) |
Group Search API.
Documentation: https://docs.gitlab.com/ee/api/search.html#group-search-api |
GET /groups/:id/search?scope=* |
| 35 | search_projects
(only EE) |
Project Search API.
Documentation: https://docs.gitlab.com/ee/api/search.html#project-search-api |
GET /projects/:id/search?scope=* |
| 36 | user | List current user.
Documentation: https://docs.gitlab.com/ee/api/users.html#list-current-user-for-normal-users |
GET /user |
| 37 | users | List users.
Documentation: https://docs.gitlab.com/ee/api/users.html#list-users |
GET /users |
Tests: Web
edit| NN | Name | Description | Endpoint |
| 1 | group | Web - Group Page.
Controllers: `GroupsController#show`, `Groups::ChildrenController#index` |
GET /:group |
| 2 | group_issues | Web - Group Issues Page.
Controllers: `GroupsController#issues` |
GET /groups/:group/issues |
| 3 | group_merge_requests | Web - Group Merge Requests Page.
Controllers: `GroupsController#merge_requests` |
GET /groups/:group/merge_requests |
| 4 | project | Web - Project Page.
Controllers: `ProjectsController#show`, `Projects::BlobController#show` |
GET /:group/:project |
| 5 | project_branches | Web - Project Branches Page.
Controllers: `BranchesController#index`, `Projects::BranchesController#diverging_commit_counts` |
GET /:group/:project/branches |
| 6 | project_commit | Web - Commit Details Page.
Controllers: `Projects::CommitController#show`, `Projects::CommitController#branches`, `Projects::CommitController#merge_requests.json` |
GET /:group/:project/commit/:commit_sha |
| 7 | project_commits | Web - Project Commits Page.
Controllers: `CommitsController#show` |
GET /:group/:project/commits/:branch |
| 8 | project_file_blame | Web - Project File Blame Page.
Controllers: `Projects::BlameController#show` |
GET /:group/:project/blame/master/:file_path |
| 9 | project_file_rendered | Web - Project File Rendered.
Controllers: `Projects::BlobController#show`, `Projects::BlobController#show.json` |
GET /:group/:project/blob/master/:file_path?viewer=rich |
| 10 | project_file_source | Web - Project File Source.
Controllers: `Projects::BlobController#show`, `Projects::BlobController#show.json` |
GET /:group/:project/blob/master/:file_path |
| 11 | project_files | Web - Project Files Tree.
Controllers: `Projects::TreeController#show`, `Projects::BlobController#show.json`, `Projects::RefsController#logs_tree.json` |
GET /:group/:project/tree/master |
| 12 | project_issue | Web - Project Issue Page.
Controllers: `Projects::IssuesController#show`, `Projects::IssuesController#discussions`, `Projects::IssuesController#related_branches,`Projects::IssuesController#can_create_branch` |
GET /:group/:project/issues/:issue_iid |
| 13 | project_issues | Web - Project Issues Page.
Controllers: `Projects::IssuesController#index` |
GET /:group/:project/issues |
| 14 | project_merge_request_changes | Web - Project Merge Request Changes Page
Controllers: `Projects::MergeRequestsController#show`, `Projects::MergeRequests::DiffsController#diffs_metadata.json`, `Projects::MergeRequests::DiffsController#diffs_batch.json` |
GET /:group/:project/merge_requests/:merge_request_iid/diffs |
| 15 | project_merge_request_commits | Web - Project Merge Request Commits Page.
Controllers: `Projects::MergeRequestsController#show`, `Projects::MergeRequestsController#commits.json` |
GET /:group/:project/merge_requests/:merge_request_iid/commits |
| 16 | project_merge_request_discussions | Web - Project Merge Request Discussions Page.
Controllers: `Projects::MergeRequestsController#show`, `Projects::MergeRequests::ContentController#widget.json`, `Projects::MergeRequestsController#discussions.json` |
GET /:group/:project/merge_requests/:merge_request_iid |
| 17 | project_merge_requests | Web - Project Merge Requests Page.
Controllers: `Projects::MergeRequestsController#index` |
GET /:group/:project/merge_requests |
| 18 | project_pipelines | Web - Project Pipelines Page.
Controllers: `Projects::PipelinesController#index`, `Projects::PipelinesController#index.json` |
GET /:group/:project/pipelines |
| 19 | project_pipelines_pipeline | Web - Project Pipeline Page.
Controllers: `Projects::PipelinesController#show`, `Projects::PipelinesController#show.json`, `Projects::Pipelines::TestsController#summary`, `Projects::PipelinesController#status` |
GET /:group/:project/pipelines/:pipeline_id` |
| 20 | project_tags | Web - Project Tags Page.
Controllers: `Projects::TagsController#index` |
GET /:group/:project/tags |
| 21 | search_global
(only EE) |
Web - Global Search
Controllers: `SearchController#show`, `SearchController#count` |
GET /:search |
| 22 | search_groups
(only EE) |
Web - Group Search
Controllers: `SearchController#show`, `SearchController#count` |
GET /:search?group_id=:id |
| 23 | search_projects
(only EE) |
Web - Projects Search
Controllers: `SearchController#show`, `SearchController#count` |
GET /:search?project_id=:id |
| 24 | User | Web - User Page
Controllers: `UsersController#show`, `UsersController#calendar.json` |
GET /:user |
Output
edit| NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT |
| web_user | 50/s | 46.19/s (>24.00/s) | 708.47ms | 957.05ms (<4000ms) | 100.00% (>99%) | Passed |
| api_v4_user | 50/s | 47.35/s (>40.00/s) | 169.85ms | 184.53ms (<500ms) | 100.00% (>99%) | Passed |
| git_push | 2/s | 1.57/s (>1.60/s) | 491.24ms | 768.68ms (<1000ms) | 100.00% (>99%) | Passed |
| git_pull | 5/s | 4.47/s (>4.00/s) | 219.39ms | 241.10ms (<500ms) | 100.00% (>99%) | Passed |