Continuous Integration#
The UltraZohm-Project uses two different build pipelines to test the builds of the UltraZohm software as well as the documentation (docs).
The Bitbucket pipeline tests & deploys the docs
Tests the build of the sphinx docs
Deploys the docs of the main branch to docs.ultrazohm.com
Mirrors the branch to ultrazohm/ultrazohm_sw
The Drone test pipeline builds UltraZohm Vivado and Vitis Project
Builds bitstream in Vivado (only main & develop branch)
Commits the vivado binarys (.xsa) to the repository and pushes the change (only on main)
Creates a new tag and a changelog, commits them to the repository and pushes the changes to bitbucket (only on main)
Exports the bitstream
Generates the Vitis workspace
Builds the software
Bitbucket pipeline (docs)#
bitbucket-pipelines.yml
configures the Bitbucket pipelinePipeline steps to build the sphinx documentation on every push to the repository (for all branches)
Pipeline steps to deploy the documentation to the UltraZohm-Server (docs.ultrazohm.com) after every merged pull request on main
Pipeline reports success or failure to Bitbucket repository (green / red symbol next to branch in Bitbucket)
Fig. 21 Setup of the Bitbucket build pipeline.#
The build pipeline:
Pulls the docker image with python
Installs the requirements for the build of the sphinx documentation
Builds the docs and treats all warnings as errors but keeps building to investigate the logs if the build fails
If the pipeline is triggered from
main
:The
build
folder aftermake html
is copied to the web serverDone with rsync deploy pipe
Variables for username, password and server path are stored as secret repository variables
Only accessible for admins:
repository settings -> repository variables
in Bitbucket (ultrazohm_sw
repository)
1# This is a sample build configuration for Python.
2# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.
3# Only use spaces to indent your .yml configuration.
4# -----
5# You can specify a custom docker image from Docker Hub as your build environment.
6image: python:3.9.7
7clone:
8 lfs: true
9
10pipelines:
11 branches:
12 # develop:
13 # - step:
14 # name: build and push docker
15 # script:
16 # - export BITBUCKET_COMMIT_SHORT="${BITBUCKET_COMMIT::7}" # https://community.atlassian.com/t5/Bitbucket-questions/BITBUCKET-COMMIT-variable-short-7digits-in-pipelines/qaq-p/752782
17 # - echo $BITBUCKET_COMMIT_SHORT
18 # - cd .devcontainer
19 # - ls
20 # - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_PASSWORD
21 # - docker build -t ultrazohm/ultrazohm_remote_container:$BITBUCKET_COMMIT_SHORT .
22 # - docker push ultrazohm/ultrazohm_remote_container:$BITBUCKET_COMMIT_SHORT
23 # services:
24 # - docker
25 # - step:
26 # name: Github-Mirror
27 # image: alpine/git
28 # script:
29 # - apk add git-lfs
30 # - git remote add mirrorOrigin git@github.com:ultrazohm/ultrazohm_sw.git
31 # - git lfs fetch --all
32 # - git push --all mirrorOrigin
33 main:
34 - step:
35 name: Deploy to Contabo Server
36 deployment: contabo
37 script:
38 - apt-get update && apt-get -y install --no-install-recommends texlive
39 - apt-get -y install --no-install-recommends texlive-latex-extra
40 - apt-get -y install --no-install-recommends texlive-lang-german
41 - apt-get -y install --no-install-recommends texlive-pictures
42 - apt-get -y install --no-install-recommends texlive-science
43 - apt-get -y install --no-install-recommends texlive-luatex
44 - apt-get -y install --no-install-recommends texlive-plain-generic
45 - apt-get -y install --no-install-recommends ghostscript
46 - apt-get -y install --no-install-recommends pdf2svg
47 - apt-get -y install --no-install-recommends doxygen
48 - apt-get -y install --no-install-recommends libpcre3
49 - apt-get -y install --no-install-recommends libpcre3-dev
50 - python -m pip install -U pip
51 - ls
52 - cd docs
53 - ls
54 - pip install -r requirements.txt
55 - make doxygen
56 - make html SPHINXOPTS="-W --keep-going -n"
57 - ls
58 - echo "Deploying to test environment"
59 - pipe: atlassian/rsync-deploy:0.4.3
60 variables:
61 USER: $ls_user
62 SERVER: $hostname
63 REMOTE_PATH: $docs_path
64 LOCAL_PATH: ./build/html/
65 # The --chmod causes the following
66 # Directories: owner: rwx, group: r-x, other: r-x
67 # Files: owner: rw-, group: r--, other: ---
68 # +X: All files, the owner already had execution right on gets executable
69 # for everybody and directories get accessable
70 EXTRA_ARGS: '-a --delete --chmod=D0755,F0640,+X'
71 SSH_PORT: '22'
72 - step:
73 name: Github-Mirror
74 image: alpine/git
75 script:
76 - apk add git-lfs
77 - git remote add mirrorOrigin git@github.com:ultrazohm/ultrazohm_sw.git
78 - git lfs fetch --all
79 - git push --all mirrorOrigin
Unit tets (Ceedling)#
All unit tests (see Unit tests (Ceedling) are run in the Bitbucket pipeline.
If one test fails, the pipeline fails.
Bitbucket pipeline (GitHub-Mirror)#
Adds Github repository as remote repo
Pushes current branch to ultrazohm/ultrazohm_sw
A special UltraZohm Github Account (Login information in Keepass) pushes to the Github repository
The account uses the Bitbucket pipeline SSH key (
ultrazohm_sw -> Repository settings --> SSH keys
, only visible to admins) to push to GithubGithub.com is added to
Known hosts
inultrazohm_sw -> Repository settings --> SSH keys
Static code check#
We use cppcheck as our static code analyser.
Useful information:
Note
The build pipeline does not fail if there are warnings from the static code analysis!
We check with regular cppcheck and with the extensions for MISRA and CERT rules. MISRA violations can be suppressed on a line-by-line basis, e.g.:
To suppress a MISRA violation, write a comment in the line before the violation // cppcheck-suppress misra-c2012-$rule_number
with $rule_number
specifying which rule the code violates.
After the rule number, add //.
and write a regular comment stating why this deviation is justified.
Warning
Do not use this lightly! The only two places we deviate so far (in new code) is the <stdio.h> for testing and read/write from AXI since the latter requires casting.
UltraZohm development container#
To reduce the time the CI steps spend on installing dependencies, we use a docker image for the UltraZohm development container.
The container is based on the VS Code Remote Container
Updating the docker image on dockerhub is a manual process
Update:
1.. Rebuild the container with VS Code locally (to also run the post-build steps!) 2. Login to docker.io on the CLI with your credentials (account has to be part of ultrazohm team on Dockerhub):
podman login docker.io
Tag the local image (podman/docker) (the name of the local image may change):
podman image tag localhost/vsc-ultrazohm_sw-ad6053a7600060d35be6bf639d2373c4:latest ultrazohm/ultrazohm_remote_container
Push the image to dockerhub
podman push ultrazohm/ultrazohm_remote_container:latest
Drone pipeline (Software)#
Drone pipeline builds
Uses Drone
Drone has a Server and a Runner
Drone Server is the bridge between Bitbucket, User and Runner (ci.ultrazohm.com)
Runner (Docker runner, see Drone docs) polls the server and executes the build pipeline
Server and Runner are used as Docker container on the UltraZohm Server
Setup in Bitbucket exactly as in the drone docs for Bitbucket
Permissions in the OAuth settings of the Bitbucket repository must match the drone docs
drone/Docker-compose.yml
in uz_server_main repository (only visible to admins) sets up DroneChanges in the docker-compose file are automatically transferred to the UltraZohm by using a rsync pipeline
ssh
to UltraZohm-Server, cd to/drone
and usedocker-compose up -d
to restart the Drone Server and Runner after changes in the repository
How-to:
ssh $username@ultrazohm.com
cd $path/drone
docker-compose up -d
Setup#
Fig. 22 Setup of the Drone pipeline#
Drone Runner and Vivado Docker image#
The Drone Runner starts a Docker Container for each pipeline step.
To use Vivado and Vitis, we use a costum Vivado docker image (vivado:2020.1
).
There are two challenges with using Vivado and Vitis in a Docker container.
Xilinx license is locked to the MAC
Create license on xilinx.com/getlicense
Lock the license to the MAC of the machine that executes the runner
The UltraZohm-Server license is locked to the Docker network
drone_default
(seedocker network ls
anddocker network inspect host
)Download the
.lic
fileFollow the instructions in uz_server_drone repository (only for admins due to licensing & login information)
Make the network adapter that is used for the license MAC-lock to the Drone Runner
See https://docs.drone.io/runner/docker/configuration/reference/drone-runner-networks/
E.g. DRONE_RUNNER_NETWORKS=drone_default
See the docker-compose file for details (drone/docker-compose.yml, only for admins due to login information in the file).
Following is an example docker-compose file without login information.
Note that the indentation is relevant since this is a yml
file.
Furthermore, some configuration is specific to the UltraZohm-Server.
version: '3.7'
services:
drone-server:
container_name: drone_server
image: drone/drone:1
volumes:
- /var/lib/drone:/data
- /var/run/docker.sock:/var/run/docker.sock
restart: always
userns_mode: "host"
environment:
- DRONE_BITBUCKET_CLIENT_ID=$bitbucket_key
- DRONE_BITBUCKET_CLIENT_SECRET=$bitbucket_secret
- DRONE_RPC_SECRET=$common_secret
- DRONE_SERVER_HOST=ci.ultrazohm.com
- DRONE_SERVER_PROTO=https
- DRONE_USER_CREATE=username:$username,admin:true
networks:
- frontend
drone-runner:
container_name: drone_runner
image: drone/drone-runner-docker:1
ports:
- "3000:3000"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
restart: always
userns_mode: "host"
environment:
- DRONE_RPC_PROTO=https
- DRONE_RPC_HOST=ci.ultrazohm.com
- DRONE_RPC_SECRET=$common_secret
- DRONE_RUNNER_CAPACITY=1
- DRONE_RUNNER_NAME=$runnerName
- DRONE_LOGS_TRACE=true
- DRONE_LOGS_PRETTY=true
- DRONE_LOGS_COLOR=true
- DRONE_RUNNER_NETWORKS=drone_default
- DRONE_CPU_SET=1,2,3,4
- DRONE_MEMORY_LIMIT=20000000000
networks:
frontend:
external:
name: frontend
Push back to repository#
The drone pipeline pushes back to the ultrazohm_sw
repository if a commit is made to main (through a pull request).
The pipeline:
commits all binarys to the repo
Creates a tag using autotag
Settings for the tag: conventional commits
Pushes back to main using
https
Creates
CHANGELOG.md
by using auto-changelogCommits the changelog
Pushes back to main using
https
The tag is on the commit of the binarys, the CHANLOG.md is one commit after the tag
Commit message is fixed and starts with [skip ci] to prevent infinite CI loop