Security/Application Security Pipeline
Purpose
editThis document provides guidance on how to implement security into the CI/CD pipeline, leveraging both GitLab's integrated tools and custom tools provided and developed by the Security Team.
In an effort to improve application security testing, our goal has been to “shift left” to remove more vulnerabilities earlier. The idea is to empower developers to find and fix vulnerabilities earlier in the software development lifecycle, when changes are less costly and more timely.
With security embedded into the development workflow, developers can get feedback on the security of their code as they are working, they can remediate in real time, and free up the security team’s time to focus on monitoring issues, assessing risk, and solving vulnerabilities that can’t be fixed by the developer. By continuously testing even small, incremental code changes, an avalanche of work is avoided at the end of the SDLC.
Note: The Security Team strongly recommends including security scanning tools into either migrated or new repository pipelines. These features have to be triggered both for new Merge Requests and for Continuous Development/Delivery.
Application Security Automation Methodologies
editTemplates Provided by the Security Team
editThe Security Team strongly suggests to use the relevant templates located at this GitLab repository.
In order to include these templates, the preferred approach would be this:
# security ci templates
include:
# osv generic template
- project: 'repos/security/gitlab-ci-security-templates'
ref: 0.1.3
file: 'generic-osv/osv-ci.yaml'
variables:
WM_APPSEC_GOLANG_IMAGE: bookworm:20231015
For further guidance please see the documentation.
Custom Templates
editEven though the Security Team strongly advises against the use of custom templates, there is the possibility to implement your own custom ones.
GitLab Default Templates
editFor all of the remaining Languages/Frameworks not supported by the Security Team, there is the possibility to use the GitLab default one. It features automatic language detection which works even for mixed-language projects. If any supported language is detected in the project source code it automatically runs the appropriate SAST analyzers.
Even though the results may not be the greatest, it could still provide some value. Here is the list of GitLab supported languages.
How to configure GitLab default SAST templates
editTo enable and configure SAST with default settings:
- On the top bar, select Menu > Projects and find your project.
- On the left sidebar, select Security & Compliance > Configuration.
- In the SAST section, select
Enable via MR. - Review the draft MR that enables SAST with the default recommended settings in the
.gitlab-ci.ymlfile. - Merge the MR to enable SAST. You should see SAST jobs run in that MR’s pipeline.
Multi-language CI Templates
editName of the template: osv-ci.yaml
Description: OSV-Scanner provides an officially supported frontend to the OSV database that connects a project’s list of dependencies with the vulnerabilities that affect them. Since the OSV.dev database is open source and distributed, it has several benefits in comparison with closed source advisory databases and scanners:
How to include it: change your .gitlab-ci.yml accordingly:
# set appsec-related template variables
variables:
WM_APPSEC_GOLANG_IMAGE: "bookworm:20231015"
include:
# osv ci template
- project: 'repos/security/gitlab-ci-security-templates'
ref: 0.1.3
file: 'generic-osv/osv-ci.yaml'
Environmental variables: These tools are triggered for each and every new push or merge request. It is also possible to trigger them manually by visiting the CI/CD section on GitLab.
WM_APPSEC_GOLANG_IMAGEthe suggested image is:bookworm:20231015WM_APPSEC_OSV_OPTIONSoverrides default arguments passed.
Expected output: The default format, which outputs the results as a human-readable table.
╭─────────────────────────────────────┬──────┬───────────┬──────────────────────────┬─────────┬────────────────────╮
│ OSV URL │ CVSS │ ECOSYSTEM │ PACKAGE │ VERSION │ SOURCE │
├─────────────────────────────────────┼──────┼───────────┼──────────────────────────┼─────────┼────────────────────┤
│ https://osv.dev/GHSA-c3h9-896r-86jm | 8.6 │ Go │ github.com/gogo/protobuf │ 1.3.1 │ path/to/go.mod │
│ https://osv.dev/GHSA-m5pq-gvj9-9vr8 | 7.5 │ crates.io │ regex │ 1.3.1 │ path/to/Cargo.lock │
╰─────────────────────────────────────┴──────┴───────────┴──────────────────────────┴─────────┴────────────────────╯
Name of the template: seeking-secrets.yaml
Description: Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.
How to include it: change your .gitlab-ci.yml accordingly:
# set appsec-related template variables
variables:
WM_APPSEC_PYTHON_IMAGE: python3-build-bullseye:0.0.1
include:
- project: repos/security/gitlab-ci-security-templates
ref: 0.1.3
file: secret-seeker/seeking-secrets.yaml
Environmental variables:
WM_APPSEC_PYTHON_IMAGE:the suggested image is:python3-build-bullseye:0.0.1
Expected Output:
commit ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29
Author: zricethezav <thisispublicanyways@gmail.com>
Date: Sun Jan 28 17:39:00 2018 -0500
[update] entropy check
diff --git a/checks_test.go b/checks_test.go
--- a/checks_test.go
+++ b/checks_test.go
@@ -28,0 +37,1 @@
+ "aws_secret= \"AKIAIMNOJVGFDXXXE4OA\"": true,
In order to use semgrep, include the relevant CI template within your .gitlab-ci.yml like any other provided security include/template.
You can configure which rule set or collection of rules to run by changing the variable WM_APPSEC_SEMGREP_OPTIONS. Not that you can chain multiple --config={some-ruleset} options to leverage multiple policies or rules. For example:
variables:
WM_APPSEC_PYTHON_IMAGE: "python3-build-bullseye:0.0.1"
# A simple configuration for a PHP application:
WM_APPSEC_SEMGREP_OPTIONS: "--config=https://semgrep-rule-merge.toolforge.org/index.php?q=php --metrics=off --time --exclude=vendor"
include:
- project: repos/security/gitlab-ci-security-templates
ref: 0.1.3
file: semgrep/semgrep-ci.yml
Note: It is always strongly recommended to use one of the available python-based docker images from https://docker-registry.wikimedia.org/.
Note: The Security Team discourages the use of --config=auto because metrics will be reported back to Semgrep.
Environmental variables:
WM_APPSEC_PYTHON_IMAGE:the suggested image is:python3-build-bullseye:0.0.1
Default Rule Sets
editUnfortunately, due to potential licensing issues with Semgrep's primarmy rule repository (see also: T304737) we cannot currently take advantage of any default rules or policies hosted at https://semgrep.dev/r, which are typically referenced via --config=p/some-policy.
Semgrep Merge Tool
editThe Semgrep Merge Tool (hosted on toolforge) is a tool developed and maintained by the Wikimedia Security Team for use with the semgrep CI template. This tool offers a curated group of OSI- and free-culture-compliant rules and policies which can be referenced via the URLs linked upon the index page. For additional usage questions, please contact the current project maintainers.
Note: the Semgrep Merge Tool should be considered the default method of using semgrep rules within Gitlab CI/CD.
Custom Rule Sets
editSemgrep supports the development of custom rules. The suggested way to do this is to create a new .yml file that contains your rules (either within your repository, or within the wikimedia-semgrep-rules repository), and then referencing these .yml files within the .gitlab-ci.yml file like this:
variables:
SEMGREP_OPTIONS:
--config=https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-templates/-/raw/main/semgrep/rules/custom-rules.yml
Default Scanning behaviour
editBy default, semgrep will only check altered files from any Gitlab push or merge request. In order to trigger a full scan, you must manually run the pipeline or schedule it.
Note: It is still possible to override this default behavior by setting the variable WM_APPSEC_SEMGREP_ALL_FILES to "true" within your repository's .gitlab-ci.yml file:
WM_APPSEC_SEMGREP_ALL_FILES: "true"
Javascript CI Templates
editName of the template: npm-outdated-nodejs-ci.yml
Description: This tool will check the registry to see if any (or, specific) installed packages are currently outdated.
How to include it: change your .gitlab-ci.yml accordingly:
variables:
WM_APPSEC_NPM_IMAGE: "nodejs12-devel:0.0.1"
include:
- project: repos/security/gitlab-ci-security-templates
ref: 0.1.3
file: npm-outdated/npm-outdated-nodejs-ci.yml
Environmental variables:
WM_APPSEC_NPM_IMAGE:the suggested image is:nodejs12-devel:0.0.1
WM_APPSEC_NPM_OUTDATED_OPTIONS:use this variable to specify custom flags.
Expected output: The Default output is the textual report of vulnerabilities in the Gitlab CI console output. NPM Outdated can also output in json.
PHP CI Templates
editName of the template: php-composer-outdated-ci.yml
Description: Composer Outdated is a sub-function of composer that checks for outdated dependencies.
How to include it: change your .gitlab-ci.yml accordingly:
variables:
WM_APPSEC_PHP_IMAGE: "releng/composer-php72:0.5.0-s6"
include:
- project: repos/security/gitlab-ci-security-templates
ref: 0.1.3
file: php-composer-outdated/php-composer-outdated-ci.yml
Environmental variables:
WM_APPSEC_PHP_IMAGE:the suggested image is:releng/composer-php72:0.5.0-s6WM_APPSEC_PHP_COMPOSER_OUTDATED_OPTIONS:Use this variable to specify custom flags.
Expected output: The Default output is the textual report of vulnerabilities in the Gitlab CI console output. Composer Outdated can also output in json.
Name of the template: php-security-checker-ci.yml
Description: PHP Security Checker is a command line tool that checks if your PHP application depends on PHP packages with known security vulnerabilities. It uses the Security Advisories Database behind the scenes.
How to include it: change your .gitlab-ci.yml accordingly:
variables:
WM_APPSEC_PHP_IMAGE: "releng/composer-php72:0.5.0-s6"
include:
- project: repos/security/gitlab-ci-security-templates
ref: 0.1.3
file: php-security-checker/php-security-checker-ci.yml
Environmental variables:
WM_APPSEC_PHP_IMAGE:the suggested image is:releng/composer-php72:0.5.0-s6WM_APPSEC_PHP_SECURITY_CHECKER_OPTIONS:Use this variable to specify custom flags.
Expected output: The Default output is the textual report of vulnerabilities in the Gitlab CI console output. Php Security Checker can also output in ansi, markdown, json, junit, and yaml.
Name of the template: php-phan-taint-check-ci.yml
Description: Phan is a static analyzer for PHP that prefers to minimize false-positives. Phan attempts to prove incorrectness rather than correctness. The phan-taint-check plugin is a phan plugin which performs security-related static analysis against PHP codebases, with a specific adaptor for MediaWiki codebases.
How to include it: change your .gitlab-ci.yml accordingly:
variables:
WM_APPSEC_PHP_IMAGE: "releng/composer-php80:0.0.2-s8"
include:
- project: repos/security/gitlab-ci-security-templates
ref: 0.1.3
file: php-phan-taint-check/php-phan-taint-check-ci.yml
Environmental variables:
WM_APPSEC_PHP_IMAGE:the suggested image is:releng/composer-php80:0.0.2-s8(php8+ is required for a symphony dependency of phan)WM_APPSEC_PPT_OPTIONS:Use this variable to specify custom cli flags.WM_APPSEC_PPT_BRANCH:The branch of mediawiki to clone, if analyzing a MediaWiki extension or skin. Default ismaster. At some point, it would be nice to automate this to reliably use$CI_COMMIT_BRANCH.WM_APPSEC_PPT_CODE_TYPE:The current supported MediaWiki types areextensionsandskins. For any other type of php application, set this variable to an empty string or any other value than "extensions" or "skins".WM_APPSEC_PPT_PROJECT_NAME:The project's git name or slug. Defaults to$CI_PROJECT_NAME.WM_APPSEC_PPT_FULL_PHAN_OUTPUT:Specify whether you'd like to see all phan issues (prefixed withPhan) or just those from the phan-taint-check plugin (prefixed withSecurityCheck). Defaults tofalse
Expected output: The Default output is the textual report of vulnerabilities in the Gitlab CI console output.
Golang CI Templates
editName of the template: go-mod-outdated.yml
Description: This tool provide an easy way to find outdated dependencies of your Go projects. go-mod-outdated provides a table view of the go list -u -m -json all command which lists all dependencies of a Go project and their available minor and patch updates. It also provides a way to filter indirect dependencies and dependencies without updates.
How to include it: change your .gitlab-ci.yml accordingly:
# set appsec-related template variables
variables:
WM_APPSEC_GOLANG_IMAGE: "bookworm:20231015"
include:
- project: 'repos/security/gitlab-ci-security-templates'
ref: 0.1.3
file: golang-go-mod-outdated/go-mod-outdated-ci.yml
Environmental variables: These tools are triggered for each and every new push or merge request. It is also possible to trigger them manually by visiting the CI/CD section on GitLab.
WM_APPSEC_GOLANG_IMAGE:the suggested image is:bookworm:20231015WM_APPSEC_GO_MOD_OUTDATED_OPTIONS:Use this variable to specify custom flags.
Expected output: The Default output is the table view of the json report of vulnerabilities in the Gitlab CI console output. go-mod-outdated can also output in json.
Name of the template: golang-gosec-ci.yml
Description: Inspects source code for security problems by scanning the Go AST.
How to include it: change your .gitlab-ci.yml accordingly:
# set appsec-related template variables
variables:
WM_APPSEC_GOLANG_IMAGE: "bookworm:20231015"
include:
- project: 'repos/security/gitlab-ci-security-templates'
ref: 0.1.3
file: 'golang-gosec/golang-gosec-ci.yml'
Environmental variables:
WM_APPSEC_PYTHON_IMAGE:the suggested image is:bookworm:20231015WM_APPSEC_GOLANG_GOSEC_OPTIONS:Use this variable to specify custom flags.
Expected output: The Default output is the textual report of vulnerabilities in the Gitlab CI console output. Gosec also supports json, yaml, csv, sonarqube, JUnit XML, html and golint output formats.
Python CI Templates
editName of the template: python-bandit-ci.yml
Description: Bandit is a tool designed to find common security issues within Python code. To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.
Bandit was originally developed within the OpenStack Security Project and later rehomed to PyCQA.
How to include it: change your .gitlab-ci.yml accordingly:
variables:
WM_APPSEC_PYTHON_IMAGE: "bookworm:latest"
include:
- project: repos/security/gitlab-ci-security-templates
ref: 0.1.3
file: python-bandit/python-bandit-ci.yml
Environmental variables:
WM_APPSEC_PYTHON_IMAGE:the suggested image is:bookworm:latestWM_APPSEC_PYTHON_BANDIT_OPTIONS:Use this variable to specify custom flags.
Expected output: The Default output is the textual report of vulnerabilities in the Gitlab CI console output. Bandit can also output in json.
Results
editAll projects are required to incorporate the Security Team's centralized Gitlab security templates into their .gitlab-ci.yml configuration file. In cases where the current templates do not support your project's specific programming language or platform, we encourage you to contact the Security Team to ensure that your project receives the necessary support. It's important to note that any failures generated by these security templates may not necessarily indicate insecure code, as false positives can occur. If you have any inquiries regarding the results produced by the security templates, please don't hesitate to get in touch with the security team. We are committed to enhancing the existing security templates to minimize false positives and focus on addressing critical security issues.
Furthermore, teams have the flexibility to permit security template failures without obstructing their development pipeline. We acknowledge that false positives should not impede development progress, but we do expect teams to carefully assess security pipeline failures and thoroughly review them for any significant security concerns.
Risk review and acceptance
editWhen a security template's CI job is triggered—whether on demand or upon issuing a new MR—each template is equipped with its own implementation of the allow_failure property. This property serves to either halt the pipeline in the event of findings or to permit and acknowledge the associated risk.
The behaviour of the pipeline regarding job failures is contingent upon the individual implementation of the allow_failure property within the security template. If allow_failure is set to true, the pipeline will persist in running subsequent jobs. On the opposite, setting allow_failure to false will terminate the pipeline's execution after a job failure.
When jobs are granted permission to fail (allow_failure: true), an orange warning indicates that a job has failed. Despite this warning, the pipeline is deemed successful, and the related commit is marked as passed without any warnings.
Proof of Concept
editIn this scenario, the gitlab-ci.yml file contains numerous security and non-security templates:
Upon the pipeline's activation and the completion of all jobs, we are able to identify the successful and failed ones:
In this instance, every job succeeded except for the OSV dependency check, which identified potential security dependencies, causing it to fail. In the scenario presented, based on the repository settings, if the option to merge despite the presence of failed jobs was enabled, the repository owner would essentially be acknowledging and accepting the risk of using the identified vulnerable dependencies flagged by the OSV tool:
Note: Owners of repositories are automatically assuming the responsibility for the findings generated by various security tools executed within Gitlab CI. As a result, it falls upon them to either resolve the identified issues or acknowledge and assume the associated risk. If assistance is needed in addressing these issues, we encourage reaching out to the Security Team for guidance and support.
How to check results
editIn order to check the output of various security tools, please go to the Jobs section of your repository, under the CI/CD > Pipelines section of your repository. Make sure you are viewing the correct branch of a repository as well.
Based on the specific tool configuration, a job could never fail, or fail only when new critical/high/moderate vulnerabilities are discovered. For example, the composer outdated and npm outdated are set to allow failures within a given pipeline, implying they are more informational in nature and shouldn't block any merge or deployment. Whereas as semgrep and other tools would fail within a pipeline, with the expectation that a code maintainer would work through various reported issues and either resolve them or report them as false positives and possibly suppress future output with various mechanisms supported by an individual tool.
JSON Artifacts
editSometimes security tools emit JSON output in the form of a report. The JSON report file can be downloaded from the CI pipelines page, or the pipelines tab on merge requests by setting artifacts: paths to gl-sast-report.json. For more information see the documentation and [1]. This can be a more secure and customizable approach to security reports, but as nearly all Wikimedia repos, merge requests, etc. are or will be public, there isn't much advantage to this approach given current realities.
Example
For phpcs-security-tool add this to the gitlab-ci.yml file:
phpcs-security-audit-sast:
artifacts:
paths:
- gl-sast-report.json