Summary[edit]

1	Project Name	GitLab: Ansible Installation and Performance tests
2	URL	TEST: https://gitlab-test.wmcloud.org PROD: https://gitlab.wikimedia.org
3	Audience	Anonymous developers Authenticated developers Bots
4	Countries targeted	World
5	Language(s) (ISO 639-1)	en
6	Development Type	New project (with possible reusing elements from previous)
7	Ticket system	Phabricator -- https://phabricator.wikimedia.org (see the workboard https://phabricator.wikimedia.org/project/view/5212/) Redmine -- https://redmine.gluzdov.com
8	Delivery timeline and status	Initial Collaboration Testing Setup Building Ansible playbook for GitLab Server Building Ansible playbook for GitLab Runner Performance Testing MVP Delivering

Introduction[edit]

The GitLab Initialization project aims to provide WikiMedia Foundation’s (WMF) Release Engineering team (RelEng) with a minimum viable product (MVP) type GitLab installation.

GitLab is a version control and Continuous Integration (CI) system. It is a web application written in Ruby on Rails accessing Postgresql and Redis as databases and other shared storage to store the source code repositories. There is an Enterprise Edition (EE) which is licensed software and a Community Edition (CE) which is open source and free edition. We will be using GitLab CE.

Currently WMF uses a combination of tools including gerrit/zuul/jenkins in a similar function. The usage is about 12 gitops per sec, 1300 repositories, 800 GB size, 1000s users, 250 active/month, about 30 CI VMs with 450,000 CI minutes/month, so 43800 min/month, approximately 11 servers active at any given point in time. CI VMs run under WikiMedia Cloud Services (WMCS).

The current gerrit/zuul/jenkins system has a limited high availability setup with warm standbys in an alternate data center that could be activated within hours. The CI VMs however run on WMCS which is only available in the eqiad data center. We have not tested a failover of the systems.

Under the MVP approach RelEng plans to transfer a number of WMF repositories, starting with a subset of RelEng's repositories and working with volunteer WMF teams.

The MVP installation is oriented towards minimal change from the current setup in terms of infrastructure. In terms of function the MVP setup is affording us the ability to gain experience with being good project owners of a GitLab install: maintenance tasks that must be automated, production systems integrations, ensure we can run a testing environment that matches a production environment, explore and establish login and authentication options, and establish deployment and upgrade cadence. These are all part of the setup phase (https://www.mediawiki.org/wiki/GitLab/Roadmap#2._Setup) of the GitLab roadmap.

A team of consultants from Speed&Function has been contracted to perform the MVP installation. See T274458 as the main entry point in phabricator.

The Service Operations team in Site Reliability Engineering (SRE) is hiring 2 SREs that will be responsible for the GitLab infrastructure. These SREs will be part of a FY 21/22 project to review the GitLab installation and reinstall GitLab according to the review. The team of consultants is tasked with providing documentation necessary for the review and reinstallation.

Installation parameters[edit]

Network position[edit]

GitLab MVP is installed in the production network due to access and data storage requirements. To avoid a dependency between the traffic infrastructure and GitLab MVP it is not behind the traffic infrastructure but has its own external IP addresses, similar to gerrit, a machine address and a service address. A test installation of GitLab is running in WMCS. (T278197)

Compute Infrastructure[edit]

GitLab MVP runs on a single standalone server in the production network running a Debian based distribution. Performance tests have shown that a single server is capable of handling WMF’s traffic and processing needs. The server is managed with Puppet: firewall, certificates, OS packages etc. The GitLab MVP application is managed by Ansible for installation and configuration. High-availability considerations will be addressed in the upcoming GitLab reinstallation project.

Storage Infrastructure[edit]

GitLab MVP uses internal storage for all purposes.

GitLab Application Installation[edit]

GitLab is installed through GitLab provided omnibus packages and configured through the gitlab.rb file. HTTPS is implemented through the included nginx server, while certificates are managed through WMF puppet. GitLab git via ssh access is implemented through a secondary SSH server on port 22 on the service IP address. CI/CD integration runs on GitLab runners in WMCS. GitLab runner location considerations will be addressed in the upcoming GitLab reinstallation project.

Authentication[edit]

GitLab MVP will use WMF CAS-SSO Apereo Single Signon. Group integration is not available in GitLab CE and will be done manually and automated in a later effort by WMF.

Monitoring[edit]

The GitLab MVP host is monitored via Prometheus and Icinga for OS infrastructure parameters. Dashboards for OS and GitLab are available in Grafana. Basic uptime checks are in Icinga.

Logging[edit]

GitLab MVP logs everything to /var/log/gitlab/ directory. Files are mirrored to the foundation’s logstash server.

Backup[edit]

GitLab MVP uses GitLab built-in backup routines to generate backups via cron. There is daily full backup of everything but build artifacts and logs. Backups are stored locally. WMF bacula picks up backup files once a day.

Security and Updates[edit]

OS security is the responsibility of the SRE team.

GitLab security is the responsibility of the Release Engineering team.

Architecture of the Performance tests[edit]

The GitLab Performance Tool (GPT) is built and maintained by the GitLab Quality Enablement team to provide performance testing of any GitLab instance. The tool has been built upon the open source tool K6.io and provides numerous tests that are designed to effectively test gitlab.

The tool can be used both manually and automatically, with us doing the latter for automated testing of reference environments via Pipelines.

GPT is used by the GitLab Quality Enablement team to continuously perform test GitLab on environments based on our Reference Architectures that have been built with the GitLab Environment Toolkit. For more information please refer to our blog post - How our QA team leverages GitLab’s performance testing tool (and you can too).

The purpose of the performance test is to see whether the configured VM instance for GitLab is capable of delivering the transaction rate specified, i.e. about 12 gitops per sec on 1300 repositories with a 800 GB size. Thousands of users with about 250 active/month.

The performance of a VM with 8 processors and 16 GB of memory was adequate in our tests on WMCS. Tests in production on a VM with 8 cores and 12 GB are pending.

Test Scenarios and outputs[edit]

Git
API
WEB
Output

See Appendices

GitLab automated configuration[edit]

Ansible playbook[edit]

Ansible is used to set up and configure Gitlab Server and Gitlab Runner. Ansible was selected to avoid conflict with foundation’s puppet configuration management system, while still documenting and automating the setup. The following components are installed via Ansible:

The gitlab application and its configuration
The secondary SSH server to be used for git
The gitlab runner responsible for CI/CD jobs

GitLab CI is a powerful tool that can be used for a number of things, including infrastructure as code, deployments and GitOps.

Before running Ansible playbook it needs checking the ability of ssh-ing the target hosts and have sudo permissions there

Automated GitLab Server installation[edit]

GitLab (as a platform) is installed and configured with provided Ansible playbook which can be found on WMF's Gerrit: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible/
Secondary SSH server for git
Hosts to deploy GitLab Servers need to be configured in [gitlab_servers] section of 'hosts' file (it's an Ansible inventory configuration file)
Ansible playbook parameters are set up on host-by-host basis in host_vars/<hostname> YAML files
List of available playbook parameters can be found in gitlab_server/defaults/main.yml file
To run Ansible playbook just execute install-gitlab-server.sh script

Automated GitLab Runner installation[edit]

GitLab Runner is installed and configured with provided Ansible playbook which can be found on WMF's Gerrit: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/gitlab-ansible/
Hosts to deploy GitLab Runners need to be configured in [gitlab_runners] section of 'hosts' file (it's an Ansible inventory configuration file)
Ansible playbook parameters are set up on host-by-host basis in host_vars/<hostname> YAML files
List of available playbook parameters can be found in gitlab_runner/defaults/main.yml file
To run Ansible playbook, execute install-gitlab-runner.sh script

Appendices[edit]

A: Decision Log[edit]

Link

B: Tests: Scenarios[edit]

NN	Name	Description	Endpoint
1	api_list_group_variables	List group variables Setup stage: Create group and multiple group variables Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/group_level_variables.html#list-group-variables	GET /groups/:id/variables
2	api_list_project_variables	List project variables Setup stage: Create group, project and multiple project variables Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/project_level_variables.html#list-project-variables	GET /projects/:id/variables
3	api_new_branches	Create a new branch in the repository Setup stage: Create group and project Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/branches.html#create-repository-branch	POST /projects/:id/repository/branches
4	api_new_commits	Create a commit with multiple files and actions Setup stage: Create group and project Teardown stage: Delete group Dicumentation: https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions	POST /projects/:id/repository/commits
5	api_new_group_variables	Creates a new group variable Setup stage: Create group Teardown stage: Delete group Documentation https://docs.gitlab.com/ee/api/group_level_variables.html#create-variable:	POST /groups/:id/variables
6	api_new_issues	Creates a new project issue Setup stage: Create group and project Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/issues.html#new-issue	POST /projects/:id/issues
7	api_new_project_variables	Creates a new project variable Setup stage: Create group and project Teardown stage: Delete group Documentation: https://docs.gitlab.com/ee/api/project_level_variables.html#create-variable	POST /projects/:id/variables

Tests: Git[edit]

NN	Name	Description	Endpoint
1	ls_remote	Git Refs List via HTTPS Controllers: Repositories::GitHttpController#info_refs	GET /:group/:project.git/info/refs?service=git-upload-pack
2	pull	Git Pull via HTTPS to pull from master having another branch locally. Documentation: https://gitlab.com/gitlab-org/quality/performance/-/blob/master/docs/test_docs/git_pull.md	GET /:group/:project.git/info/refs?service=git-upload-pack POST /:group/:project.git/git-upload-pack
3	push	Git push commit(s) via HTTPS. Documentation: https://gitlab.com/gitlab-org/quality/performance/-/blob/master/docs/test_docs/git_push.md	GET /:group/:project.git/info/refs?service=git-receive-pack POST /:group/:project.git/git-receive-pack
4	pull_bulk	Git BULK Pull via HTTPS to pull from master having another branch locally. Documentation: https://gitlab.com/gitlab-org/q uality/performance/-/blob/master/docs/test_docs/git_pull.m	GET /:group/:project.git/info/refs?service=git-upload-pack POST /:group/:project.git/git-upload-pack

Tests: API[edit]

NN	Name	Description	Endpoint
1	groups	List groups. Documentation: https://docs.gitlab.com/ee/api/groups.html#list-groups	GET /groups
2	groups_group	Get all details of a group. Documentation: https://docs.gitlab.com/ee/api/groups.html#details-of-a-group	GET /groups/:id
3	groups_group_subgroups	List a group’s subgroups. Documentation: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-subgroups	GET /groups/:id/subgroups
4	groups_issues	List groups issues. Documentation: https://docs.gitlab.com/ee/api/issues.html#list-group-issues	GET /groups/:id/issues
5	groups_merge_requests	List groups merge requests. Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#list-group-merge-requests	GET /groups/:id/merge_requests
6	groups_projects	Get a list of projects in this group. Documentation: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-projects	GET /groups/:id/projects
7	projects	Get a list of all projects. Documentation: https://docs.gitlab.com/ee/api/projects.html#list-all-projects	GET /projects?order_by=id&sort=asc GET /projects?pagination=keyset&order_by=id&sort=asc
8	projects_deploy_keys	Get a list of a project’s deploy keys. Documentation: https://docs.gitlab.com/ee/api/deploy_keys.html#list-project-deploy-keys	GET /projects/:id/deploy_keys
9	projects_issues	List project issues. Documentation: https://docs.gitlab.com/ee/api/issues.html#list-project-issues	GET /projects/:id/issues
10	projects_issues_issue	Get a single project issue. Documentation: https://docs.gitlab.com/ee/api/issues.html#single-issue	GET /projects/:id/issues/:issue_iid
11	projects_languages	Get languages used in a project with percentage value. Documentation: https://docs.gitlab.com/ee/api/projects.html#languages	GET /projects/:id/languages
12	projects_merge_requests	Get all merge requests for this project. Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#list-project-merge-requests	GET /projects/:id/merge_requests
13	projects_merge_requests_merge_request	Get information about a single merge request. Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr	GET /projects/:id/merge_requests/:merge_request_iid
14	projects_merge_requests_merge_request_changes	Get single MR changes. Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr-changes	GET /projects/:id/merge_requests/:merge_request_iid/changes
15	projects_merge_requests_merge_request_commits	Get a list of merge request commits. Documentation: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr-commits	GET /projects/:id/merge_requests/:merge_request_iid/commits
16	projects_merge_requests_merge_request_discussions	Gets a list of all discussion items for a single merge request. Documentation: https://docs.gitlab.com/ee/api/discussions.html#list-project-merge-request-discussion-items	GET /projects/:id/merge_requests/:merge_request_iid/discussions
17	projects_project	Get single project. Documentation: https://docs.gitlab.com/ee/api/projects.html#get-single-project	GET /projects/:id
18	projects_project_pipelines	List project pipelines. Documentation: https://docs.gitlab.com/ee/api/pipelines.html#list-project-pipelines	GET /projects/:id/pipelines
19	projects_project_pipelines_pipeline	Get a single pipeline. Documentation: https://docs.gitlab.com/ee/api/pipelines.html#get-a-single-pipeline	GET /projects/:id/pipelines/:pipeline_id
20	projects_project_services	List project services. Documentation: https://docs.gitlab.com/ee/api/services.html#list-all-available-services	GET /projects/:id/services
21	projects_releases	Get project releases. Documentation: https://docs.gitlab.com/ee/api/releases/#list-releases	GET /projects/:id/releases
22	projects_repository_branches	Get a list of repository branches from a project, sorted by name alphabetically. Documentation: https://docs.gitlab.com/ee/api/branches.html#list-repository-branches	GET /projects/:id/repository/branches
23	projects_repository_branches_branch	Get a single project repository branch. Documentation: https://docs.gitlab.com/ee/api/branches.html#get-single-repository-branch	GET /projects/:id/repository/branches/:branch
24	projects_repository_commits	Get a list of repository commits in a project. Documentation: https://docs.gitlab.com/ee/api/commits.html#list-repository-commits	GET /projects/:id/repository/commits
25	projects_repository_commits_commit	Get a specific commit identified by the commit hash. Documentation: https://docs.gitlab.com/ee/api/commits.html#get-a-single-commit	GET /projects/:id/repository/commits/:sha
26	projects_repository_commits_commit_diff	Get the diff of a commit in a project. Documentation: https://docs.gitlab.com/ee/api/commits.html#get-the-diff-of-a-commit	GET /projects/:id/repository/commits/:sha/diff
27	projects_repository_compare_commits	Compare commits. Documentation: https://docs.gitlab.com/ee/api/repositories.html#compare-branches-tags-or-commits	GET /projects/:id/repository/compare?from=commit_sha1&to=commit_sha2
28	projects_repository_files_file	Get information about file in repository. Documentation: https://docs.gitlab.com/ee/api/repository_files.html#get-file-from-repository	GET /projects/:id/repository/files/:file_path
29	projects_repository_files_file_blame	Get blame information about file in repository. Documentation: https://docs.gitlab.com/ee/api/repository_files.html#get-file-blame-from-repository	GET /projects/:id/repository/files/:file_path/blame
30	projects_repository_files_file_raw	Get raw file from repository. Documentation: https://docs.gitlab.com/ee/api/repository_files.html#get-raw-file-from-repository	GET /projects/:id/repository/files/:file_path/raw?ref=master
31	projects_repository_tags	Get a list of repository tags in a project. Documentation: https://docs.gitlab.com/ee/api/tags.html#list-project-repository-tags	GET /projects/:id/repository/tags
32	projects_repository_tree	Get a list of repository files and directories in a project. Documentation: https://docs.gitlab.com/ee/api/repositories.html#list-repository-tree	GET /projects/:id/repository/tree
33	search_global (only EE)	Global Search API. Documentation: https://docs.gitlab.com/ee/api/search.html#global-search-api	GET /search?scope=*
34	search_groups (only EE)	Group Search API. Documentation: https://docs.gitlab.com/ee/api/search.html#group-search-api	GET /groups/:id/search?scope=*
35	search_projects (only EE)	Project Search API. Documentation: https://docs.gitlab.com/ee/api/search.html#project-search-api	GET /projects/:id/search?scope=*
36	user	List current user. Documentation: https://docs.gitlab.com/ee/api/users.html#list-current-user-for-normal-users	GET /user
37	users	List users. Documentation: https://docs.gitlab.com/ee/api/users.html#list-users	GET /users

Tests: Web[edit]

NN	Name	Description	Endpoint
1	group	Web - Group Page. Controllers: `GroupsController#show`, `Groups::ChildrenController#index`	GET /:group
2	group_issues	Web - Group Issues Page. Controllers: `GroupsController#issues`	GET /groups/:group/issues
3	group_merge_requests	Web - Group Merge Requests Page. Controllers: `GroupsController#merge_requests`	GET /groups/:group/merge_requests
4	project	Web - Project Page. Controllers: `ProjectsController#show`, `Projects::BlobController#show`	GET /:group/:project
5	project_branches	Web - Project Branches Page. Controllers: `BranchesController#index`, `Projects::BranchesController#diverging_commit_counts`	GET /:group/:project/branches
6	project_commit	Web - Commit Details Page. Controllers: `Projects::CommitController#show`, `Projects::CommitController#branches`, `Projects::CommitController#merge_requests.json`	GET /:group/:project/commit/:commit_sha
7	project_commits	Web - Project Commits Page. Controllers: `CommitsController#show`	GET /:group/:project/commits/:branch
8	project_file_blame	Web - Project File Blame Page. Controllers: `Projects::BlameController#show`	GET /:group/:project/blame/master/:file_path
9	project_file_rendered	Web - Project File Rendered. Controllers: `Projects::BlobController#show`, `Projects::BlobController#show.json`	GET /:group/:project/blob/master/:file_path?viewer=rich
10	project_file_source	Web - Project File Source. Controllers: `Projects::BlobController#show`, `Projects::BlobController#show.json`	GET /:group/:project/blob/master/:file_path
11	project_files	Web - Project Files Tree. Controllers: `Projects::TreeController#show`, `Projects::BlobController#show.json`, `Projects::RefsController#logs_tree.json`	GET /:group/:project/tree/master
12	project_issue	Web - Project Issue Page. Controllers: `Projects::IssuesController#show`, `Projects::IssuesController#discussions`, `Projects::IssuesController#related_branches,`Projects::IssuesController#can_create_branch`	GET /:group/:project/issues/:issue_iid
13	project_issues	Web - Project Issues Page. Controllers: `Projects::IssuesController#index`	GET /:group/:project/issues
14	project_merge_request_changes	Web - Project Merge Request Changes Page Controllers: `Projects::MergeRequestsController#show`, `Projects::MergeRequests::DiffsController#diffs_metadata.json`, `Projects::MergeRequests::DiffsController#diffs_batch.json`	GET /:group/:project/merge_requests/:merge_request_iid/diffs
15	project_merge_request_commits	Web - Project Merge Request Commits Page. Controllers: `Projects::MergeRequestsController#show`, `Projects::MergeRequestsController#commits.json`	GET /:group/:project/merge_requests/:merge_request_iid/commits
16	project_merge_request_discussions	Web - Project Merge Request Discussions Page. Controllers: `Projects::MergeRequestsController#show`, `Projects::MergeRequests::ContentController#widget.json`, `Projects::MergeRequestsController#discussions.json`	GET /:group/:project/merge_requests/:merge_request_iid
17	project_merge_requests	Web - Project Merge Requests Page. Controllers: `Projects::MergeRequestsController#index`	GET /:group/:project/merge_requests
18	project_pipelines	Web - Project Pipelines Page. Controllers: `Projects::PipelinesController#index`, `Projects::PipelinesController#index.json`	GET /:group/:project/pipelines
19	project_pipelines_pipeline	Web - Project Pipeline Page. Controllers: `Projects::PipelinesController#show`, `Projects::PipelinesController#show.json`, `Projects::Pipelines::TestsController#summary`, `Projects::PipelinesController#status`	GET /:group/:project/pipelines/:pipeline_id`
20	project_tags	Web - Project Tags Page. Controllers: `Projects::TagsController#index`	GET /:group/:project/tags
21	search_global (only EE)	Web - Global Search Controllers: `SearchController#show`, `SearchController#count`	GET /:search
22	search_groups (only EE)	Web - Group Search Controllers: `SearchController#show`, `SearchController#count`	GET /:search?group_id=:id
23	search_projects (only EE)	Web - Projects Search Controllers: `SearchController#show`, `SearchController#count`	GET /:search?project_id=:id
24	User	Web - User Page Controllers: `UsersController#show`, `UsersController#calendar.json`	GET /:user

Output[edit]

NAME	RPS	RPS RESULT	TTFB AVG	TTFB P90	REQ STATUS	RESULT
web_user	50/s	46.19/s (>24.00/s)	708.47ms	957.05ms (<4000ms)	100.00% (>99%)	Passed
api_v4_user	50/s	47.35/s (>40.00/s)	169.85ms	184.53ms (<500ms)	100.00% (>99%)	Passed
git_push	2/s	1.57/s (>1.60/s)	491.24ms	768.68ms (<1000ms)	100.00% (>99%)	Passed
git_pull	5/s	4.47/s (>4.00/s)	219.39ms	241.10ms (<500ms)	100.00% (>99%)	Passed