GitLab/Initialization/Decision Log

Date Category Decision Why? Comments
2/23/2021 Management Keep tasks and most of communication in Phabricator, the board https://phabricator.wikimedia.org/project/view/5212/ Easier tracking
2/23/2021 Management Create documentation of the process To know why the particular decision was made and what files were changed, to be able to reproduce the steps
3/1/2021 Security SSO project provides the CAS service to other services in development. As such please only use your access for creating/updating ldap accounts. Do not change any of the CAS configuration. If there are some CAS changes you want to test please let us know here and we should be able to turn things around quite quickly. https://phabricator.wikimedia.org/T274461
3/3/2021 Tools Backups: NFS works, but machines need access to where that goes.

Bacula - local backup would be good then we could point the bacula daemon Gitlab backup has to be consistent. Put on local disc, then copied and stored for later. Will grow over time. Backups task: https://phabricator.wikimedia.org/T274463

3/4/2021 Tech solutions Prometheus metrics, Grafana dashboards, and logging to Logstash have to be a part of the current scope of work https://prnt.sc/10p3h69
3/4/2021 Tools In our current case can we consider the VM in WMCS/horizon (or potentially a new VM in WMCS/horizon) as the dev environment. Then we need a second machine in our prod environment only when a load induced split is necessary. WMCS is a real testing environment with a minimal amount of scrutiny. It is the correct environment for developing a test. At the end it makes sense to run the developed test in the production environment to see what the real behavior looks like, but development is better done in WMCS/horizon
3/4/2021 Tech solutions Starting with performance testing, the results: https://docs.google.com/spreadsheets/d/1OJmkPKsQvb3YmIaG-lw7oyzZnnGxEqkCxJSPnHD1l-0/edit#gid=389161915
3/8/2021 Tools Using GPT (GitLab Performance Tool) GPT works pretty well. We use it as a base for the test scenarios. GPT is available to run on a VM
3/9/2021 Tools Using Let's Encrypt for maintaining certificates WMF uses Let's Encrypt with its servies
3/10/2021 Tools Using k6 performance testing tool Minimizing time and effort to start (and subsequent maintenance), it may make more financial sense to pay for it rather than to expand the project scope. The difference will be fading with time, so maybe we can explore a mixed approach: start with a service now, then migrated to a self-hosted k6 instance.

Support for (many) geographically distributed load agents, this results in a more realistic numbers, but I'm not sure how much value this brings to you guys. Support for test data retention, visualization and analysis, what we briefly discussed before.

3/10/2021 Tech solutions Gitlab DNS: gitlab.wikimedia.org service alias, point to gitlab1001 https://phabricator.wikimedia.org/T276170
3/16/2021 Tools Using Ansible for installing GitLab (not Puppet, and not manually) Ansible will help us to automate the routine process of Gitlab installation and system configuration, not clashing with Puppet that WMF uses to maintain hosts configurations
3/16/2021 Tech solutions For performance testing - let's skip CAS Not to have multiple user creation
3/22/2021 Tech solutions Create a new test instance for Ansible tests Not to break current server
3/30/2021 Tech solutions Performance acceptance metrics: api_v4_users, web_project, git_push, git_pull; Whole test suite as an generalized view of the Gitlab performance These metrics are representative of important use cases and general server performance
3/30/2021 Tech solutions MediaWiki/core would be an excellent test case ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/core.git It's one of our larger repos and it is one of the more accessed-by-humans repos
4/5/2021 Strategy Gitlab configuration through Web UI (or equivalent API) is not included in work by S&F, will be handled by Release Engineering Anything that is not configured in gitlab.rb is going to be maintained by the SRE team manually at this stage; In the future configuration management through API is possible
4/5/2021 Tools gitlab.rb will be version controlled in ansible repo on S&F gitlab For the flexibility on reconfiguring on different deployments
4/5/2021 Tech solutions Testing over HTTPS is good enough (don’t have to do SSH) git over HTTPS performance is representative of expected git over SSH performance, while developing native SSH tests seems resource expensive at the moment
4/5/2021 Tech solutions If you decide to dynamically resize the gitlab instance, without restarting - don’t forget to run sudo gitlab-ctl reconfigure It will self-configure itself for new parameters
4/5/2021 Tech solutions Separate admin SSH and Gitlab SSH daemons on different IP addresses For added security: Gitlab SSH will only allow gitlab user, will be configured for higher security and performance
3/10 Tech solutions Production instance certificates are managed by acme-chef, separately from Gitlab Allows for better WMF control over certificates and monitoring (https://phabricator.wikimedia.org/T276673)
5/10/2021 Tech solutions SSO/CAS decided to be the authentication method for Gitlab CAS is a hard requirement from the SRE team and it doesn't create unacceptable functional limitations at the moment
5/4/2021 Tech solutions Backups: Daily gitlab-backup backups for MVP More frequent incremental or snapshot backups research postponed after the MVP roll out
3/23/2021 Tech solutions Mail configuration outbound email is set to use a local puppet-managed MTA, inbound email is not supported, but planned for the future
5/7/2021 Tech solutions Logging: puppet-managed Logstash ingests a list of SF-suggested logs into ElastiSearch
4/10/2021 Tech solutions GiLab’s specific SSHD should have separate host keys and separate env configuration file GilLab's specific SSHD settings should'nt incluence management SSHD
5/17/2021 Tech solutions SSO/CAS decided to be the only way of authenticating users to Gitlab Per a call with the WMF team 5/17, extra authentication methods provide little value at the moment
5/17/2021 Tech solutions SSO/CAS users are to be "internal" type of users in Gitlab terms WMF strives to treat contributors the same way as employees, not much value in this distinction
5/3/2021 Tech solutions Attributes to import from SSO on authentication: 'CN' (for username and name), 'mail' for e-mail address
6/21/2021 Tech solutions Bacula Backups: use Full backups instead of Incremental GitLab is doing full backups in a self-contained .tar arcive every day. To reduce the complexity and disk usage for Bacula and represent the actual backup strategy we decided to use daily full backups. We decided against Incremental backups, because this would introduce a technical dependency between daily GitLab backups although there is no real dependency. At a later stage we may use incremental backups to snapshot git repositories or databases, but for MVP daily full backups are the best option. See https://wikitech.wikimedia.org/wiki/GitLab for more detail.