Storing CI data on Git notes

2020-11-30T00:00:00+00:00

Extending the bare bones CI server I’ve talked about before, divoplade on Freenode suggested storing CI artifacts on Git notes, such as tarballs, binaries, logs, etc.

I’ve written a small script that will put log files and CI job data on Git notes, and make it visible on the porcelain log. It is a simple extension of the previous article:

#!/usr/bin/env bash
set -Eeuo pipefail
set -x

PREFIX='/srv/ci/vps'
mkdir -p "$PREFIX"
read -r _ SHA _ # oldrev newrev refname
FILENAME="$(date -Is)-$SHA.log"
LOGFILE="$PREFIX/$FILENAME"
exec &> >(tee -a "$LOGFILE")

echo "Starting CI job at: $(date -Is)"

finish() {
  STATUS="$?"
  printf "\n\n>>> exit status was %s\n" "$STATUS"
  echo "Finishing CI job at: $(date -Is)"
  popd
  NOTE=$(cat <<EOF
See CI logs with:
  git notes --ref=refs/notes/ci-logs show $SHA
  git notes --ref=refs/notes/ci-data show $SHA
EOF
)
  git notes --ref=refs/notes/ci-data add -f -m "$STATUS $FILENAME"
  git notes --ref=refs/notes/ci-logs add -f -F "$LOGFILE"
  git notes add -f -m "$NOTE"
  printf "\n\n>>> CI logs added as Git note."
}
trap finish EXIT

unset GIT_DIR
CLONE="$(mktemp -d)"
git clone . "$CLONE"
pushd "$CLONE"
git config --global user.email git@euandre.org
git config --global user.name 'EuAndreh CI'

./container make check site
./container make publish

plaintext

The important part is in the finish() function: - #25 stores the exit status and the generated filename separated by spaces; - #26 adds the log file in a note using the refs/notes/ci-logs ref; - #27 it adds a note to the commit saying how to see the logs.

A commit now has an attached note, and shows it whenever you look at it:

$ git show 87c57133abd8be5d7cc46afbf107f59b26066575
commit 87c57133abd8be5d7cc46afbf107f59b26066575
Author: EuAndreh <eu@euandre.org>
Date:   Wed Feb 24 21:58:28 2021 -0300

    vps/machines.scm: Change path to cronjob files

Notes:
    See CI logs with:
      git notes --ref=refs/notes/ci-logs show 87c57133abd8be5d7cc46afbf107f59b26066575
      git notes --ref=refs/notes/ci-data show 87c57133abd8be5d7cc46afbf107f59b26066575

diff --git a/servers/vps/machines.scm b/servers/vps/machines.scm
index d1830ca..a4ccde7 100644
--- a/servers/vps/machines.scm
+++ b/servers/vps/machines.scm
@@ -262,8 +262,8 @@ pki " mail-domain " key  \"" (tls-priv-for mail-domain) "\""))
             (service mcron-service-type
                      (mcron-configuration
                       (jobs
-                       (list #~(job "30 1 * * 1" "guix gc -d")
-                             #~(job "30 0 * * *" "/var/lib/euandreh/backup.sh")))))
+                       (list #~(job "30 1 * * 1" "/opt/bin/gc.sh")
+                             #~(job "30 0 * * *" "/opt/bin/backup.sh")))))
             (service dhcp-client-service-type)
             #;
             (service opensmtpd-service-type

plaintext

Other tools such as cgit will also show notes on the web interface: https://euandre.org/git/servers/commit?id=87c57133abd8be5d7cc46afbf107f59b26066575.

You can go even further: since cgit can serve raw blob directly, you can even serve such artifacts (log files, release artifacts, binaries) from cgit itself:

1
2
3

$ SHA="$(git notes --ref=refs/notes/ci-logs list 87c57133abd8be5d7cc46afbf107f59b26066575)"
$ echo "https://euandre.org/git/servers/blob?id=$SHA"
https://euandre.org/git/servers/blob?id=1707a97bae24e3864fe7943f8dda6d01c294fb5c

plaintext

And like that you’ll have cgit serving the artifacts for you: https://euandre.org/git/servers/blob?id=1707a97bae24e3864fe7943f8dda6d01c294fb5c.

DIY bare bones CI server with Bash and Nix

2020-11-12T00:00:00+00:00

With a server with Nix installed (no need for NixOS), you can leverage its build isolation for running CI jobs by adding a post-receive Git hook to the server.

In most of my project I like to keep a test attribute which runs the test with nix-build -A test. This way, a post-receive hook could look like:

#!/usr/bin/env bash
set -Eeuo pipefail
set -x

LOGS_DIR="/data/static/ci-logs/libedn"
mkdir -p "$LOGS_DIR"
LOGFILE="${LOGS_DIR}/$(date -Is)-$(git rev-parse master).log"
exec &> >(tee -a "${LOGFILE}")

unset GIT_DIR
CLONE="$(mktemp -d)"
git clone . "$CLONE"
pushd "$CLONE"

finish() {
  printf "\n\n>>> exit status was %s\n" "$?"
}
trap finish EXIT

nix-build -A test

plaintext

We initially (lines #5 to #8) create a log file, named after when the run is running and for which commit it is running for. The exec and tee combo allows the output of the script to go both to stdout and the log file. This makes the logs output show up when you do a git push.

Lines #10 to #13 create a fresh clone of the repository and line #20 runs the test command.

After using a similar post-receive hook for a while, I now even generate a simple HTML file to make the logs available (example project) through the browser.

Upsides

No vendor lock-in, as all you need is a server with Nix installed.

And if you pin the Nixpkgs version you’re using, this very simple setup yields extremely sandboxed runs on a very hermetic environment.

Downsides

Besides the many missing shiny features of this very simplistic CI, nix-build can be very resource intensive. Specifically, it consumes too much memory. So if it has to download too many things, or the build closure gets too big, the server might very well run out of memory.

EuAndreh's website

Storing CI data on Git notes

DIY bare bones CI server with Bash and Nix

Upsides

Downsides