Pipeline Templates

GitLab CI pipeline templates for use in CorSAIR-standardized projects and beyond.

Setup

Makefile

Including some job templates requires that the project contain a Makefile at the root of the project repository which specifies specific targets:

# Code Quality: pre-commit hooks
# '/.gitlab/ci/stages/code_quality/jobs/pre-commit-hooks.yml'
pre-commit:
...

# SAST: dependency scanning
# '/.gitlab/ci/stages/sast/jobs/scan-dependencies.yml'
scan-dependencies:
...

# Test: release-candidate (Python)/ node:10 (Node)  
# '/.gitlab/ci/stages/test/jobs/python/release-candidate.yml'
# '/.gitlab/ci/stages/test/jobs/node/10.yml'
test:
...

# Test: python:3.7 (Python)  
# '/.gitlab/ci/stages/test/jobs/python/3.7.yml'
test-py37:
...

# Test: python:3.8 (Python)  
# '/.gitlab/ci/stages/test/jobs/python/3.8.yml'
test-py38:
...

# Test: python:3.9 (Python)  
# '/.gitlab/ci/stages/test/jobs/python/3.9.yml'
test-py39:
...

# Release: detect-and-tag-new-version
# '/.gitlab/ci/stages/release/jobs/detect-and-tag-new-version.yml'
get-project-version-number:
...

Dockerfile

If you are building and pushing a Docker image, the project must contain either:

• a Dockerfile at the root of the project repository (standard)
• a Dockerfiles directory containing one or more Dockerfiles
  • Dockerfile names will be used as image tag prefixes.
  • If any Dockerfiles depend on images built from another Dockerfile in thesame directory, to ensure correctness Dockerfiles must be named such that their sorted order iteration produces a topological ordering of container image dependencies; in other words, base images must be built prior to corresponding dependent images.

Additionally, you must supply the below required environment variables (e.g., via GitLab CI/CD UI):

1. DOCKER_NAMESPACE
2. DOCKER_PASSWORD

Usage

At the root of your project, create a .gitlab-ci.yml file that includes desired jobs and references to their stages in execution order, e.g.,:

include:
    # Note: `globals.yaml` is a mandatory inclusion.
  - project: 'CorSAIR/ci-cd/pipeline-templates'
    ref: master # Can also use Git SHAs or tags
    file: '/.gitlab/ci/stages/globals.yml'
  - project: 'CorSAIR/ci-cd/pipeline-templates'
    ref: master
    # yamllint disable-line rule:line-length
    file: '/.gitlab/ci/stages/build_and_push_docker_image/jobs/project-root-dockerfile.yml'
  - project: 'CorSAIR/ci-cd/pipeline-templates'
    ref: master
    file: '/.gitlab/ci/stages/pull_docker_image/jobs/pull.yml'
  - project: 'CorSAIR/ci-cd/pipeline-templates'
    ref: master
    file: '/.gitlab/ci/stages/sast/jobs/all.yml'
  - project: 'CorSAIR/ci-cd/pipeline-templates'
    ref: master
    file: '/.gitlab/ci/stages/sast_and_code_quality/jobs/semgrep-python.yml'
  - project: 'CorSAIR/ci-cd/pipeline-templates'
    ref: master
    file: '/.gitlab/ci/stages/code_quality/jobs/pre-commit-hooks.yml'
  - project: 'CorSAIR/ci-cd/pipeline-templates'
    ref: master
    file: '/.gitlab/ci/stages/test/jobs/python/all.yml'

stages:
  # Optional; precedes `Pull Docker Image` stage.
  - Build & Push Docker Image
  # Externally-provided required variables (e.g., via GitLab CI/CD UI):
  # 1. DOCKER_NAMESPACE
  # 2. DOCKER_PASSWORD
  #
  # Available Job configurations:
  # 1. `project-root-dockerfile.yml: project root Dockerfile build (standard).
  # 2. `dockerfiles-dir-filename-namespaced-images.yml`: multiple Dockerfiles
  #     build. Dockerfiles must be located in a project root `Dockerfiles/`  
  #     directory.

  # Mandatory; precedes remaining stages
  - Pull Docker Image
  # Remaining stages run in parallel and can be specified in any order.
  - SAST
  - SAST & Code Quality
  - Code Quality
  - Test

# Override included variables to customize your pipeline
variables:
  CI_IMAGE: <IMAGE_TO_USE_FOR_YOUR_ENTIRE_PIPELINE>
  EXTRA_VAR: <SOME_EXTRA_VAR>