gitlab CI pipeline
Recently gitlab platform becomes more and more popular. Not only does it provide the git version control, but also has embedded lots of useful devops related functionality. One of those is the option to build CI/CD pipelines, which is extremely useful for many projects. It allows you to automate tests and deployments. There are plenty of tutorials online on how to set such a pipeline up, including the official webpage. However, I want to show here how to build a more advanced CI pipeline, using different Docker images for different steps with conditions.
My custom Docker image takes some time to build, and for that reason I want to do it only when necessary. But when it is rebuild, I want to ensure that my tests are always run in proper containers.
I want to run unit tests with every branch commit. On top of that I want to execute regression tests when merge request is being created.
HOW TO
Let’s assume following:
- all jobs are run in custom Docker containers
- Docker image is rebuilt only when some changes to Dockerfile occur
- images are tagged with git branch names for ease of tracing
To make sure image is build only if needed, the first job needs to be defined with a rule tracing file changes. The image is tagged using CI_COMMIT_REF_NAME gitlab variable, which will be useful for next steps.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
docker-build: image: docker stage: build # only build when Dockerfile changed rules: - changes: - Dockerfile services: - docker:dind # log in to image registry before_script: - echo $CI_BUILD_TOKEN | docker login -u "$CI_REGISTRY_USER" \ --password-stdin $CI_REGISTRY # build and push the image script: - docker build --pull -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME \ -f Dockerfile . - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME |
Now I want to run unit tests with each commit to the branch, but using the newly created image in case commit changed the environment. Otherwise I want to use the existing latest image. I also want to ensure that if I add another commit to the branch (not changing the Dockerfile), tests still will execute on this newly-created image.
To do so I add an intermediate job to check whether branch related image exists (that’s where CI_COMMIT_REF_NAME comes in handy). As I want to enforce that this check is performed also when merge request is created, I define an if-when rule. Otherwise it would have run just once.
Job checks whether image for the branch was created (which only happens if Dockerfile was changed) and creates an artifact containing the name of the version of the image to use further.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
docker-tag: image: docker stage: tag # ensure that job is triggered with every commit rules: - if: '$CI_PIPELINE_SOURCE != "merge_request_event"' when: always - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' when: always # store version of the image to use; branch name if exists, otherwise latest script: - | if docker image inspect $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME >/dev/null; then export CURRENT_TAG=$CI_COMMIT_REF_NAME echo "CURRENT_TAG=$CI_COMMIT_REF_NAME" >> tag.env else echo "CURRENT_TAG=latest" >> tag.env fi artifacts: reports: dotenv: tag.env |
Because this job creates an artifact, now my unit tests job can depend on it.
|
1 2 3 4 5 6 7 8 |
unit-test: # use proper image image: $CI_REGISTRY_IMAGE:$CURRENT_TAG stage: test script: - # execute unit tests script dependencies: - docker-tag |
Tests will be run either on branch related image (if exists) or the current latest image.
Now for the merge request event I can run additional tests (like regression tests), using the same proper image.
|
1 2 3 4 5 6 7 8 9 10 11 |
regression-test: # use proper image image: $CI_REGISTRY_IMAGE:$CURRENT_TAG stage: test # run for merge request only rules: - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' script: - # execute regression tests script dependencies: - docker-tag |
Lastly, let’s exchange the current latest image, if new one was created, after the code merge.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
update-docker-latest: image: docker stage: deploy # run only on master branch in case there was a Dockerfile update rules: - if: '$CI_COMMIT_REF_NAME == "master" && $CURRENT_TAG != "latest"' before_script: - echo $CI_BUILD_TOKEN | docker login -u "$CI_REGISTRY_USER" \ --password-stdin $CI_REGISTRY # tag current latest image with pipeline id script: - | - docker pull $CI_REGISTRY_IMAGE:latest - docker image tag $CI_REGISTRY_IMAGE:latest $CI_REGISTRY_IMAGE:$CI_PIPELINE_ID - docker push $CI_REGISTRY_IMAGE:$CI_PIPELINE_ID # rebuild the image - docker build --pull -t $CI_REGISTRY_IMAGE -f Dockerfile . - docker push $CI_REGISTRY_IMAGE dependencies: - docker-tag |
Now let’s put all the pieces together, including the definition of the CURRENT_TAG variable.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 |
stages: - build - tag - test - deploy variables: CURRENT_TAG: "latest" docker-build: image: docker stage: build # only build when Dockerfile changed rules: - changes: - Dockerfile services: - docker:dind # log in to image registry before_script: - echo $CI_BUILD_TOKEN | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY # build and push the image script: - docker build --pull -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME -f Dockerfile . - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME docker-tag: image: docker stage: tag # ensure that job is triggered with every commit rules: - if: '$CI_PIPELINE_SOURCE != "merge_request_event"' when: always - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' when: always # store version of the image to use; branch name if exists, otherwise latest script: - | if docker image inspect $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME >/dev/null; then export CURRENT_TAG=$CI_COMMIT_REF_NAME echo "CURRENT_TAG=$CI_COMMIT_REF_NAME" >> tag.env else echo "CURRENT_TAG=latest" >> tag.env fi artifacts: reports: dotenv: tag.env unit-test: # use proper image image: $CI_REGISTRY_IMAGE:$CURRENT_TAG stage: test script: - # execute unit tests script dependencies: - docker-tag regression-test: # use proper image image: $CI_REGISTRY_IMAGE:$CURRENT_TAG stage: test # run for merge request only rules: - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' script: - # run regression tests script dependencies: - docker-tag update-docker-latest: image: docker stage: deploy # run only on master branch in case there was a Dockerfile update rules: - if: '$CI_COMMIT_REF_NAME == "master" && $CURRENT_TAG != "latest"' before_script: - echo $CI_BUILD_TOKEN | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY # tag current latest image with pipeline id script: - | - docker pull $CI_REGISTRY_IMAGE:latest - docker image tag $CI_REGISTRY_IMAGE:latest $CI_REGISTRY_IMAGE:$CI_PIPELINE_ID - docker push $CI_REGISTRY_IMAGE:$CI_PIPELINE_ID # rebuild the image - docker build --pull -t $CI_REGISTRY_IMAGE -f Dockerfile . - docker push $CI_REGISTRY_IMAGE dependencies: - docker-tag |