Building a Clojure CI/CD pipeline of CERTAIN DOOM

When things go wrong

Devops is one of those important positions where nobody cares what you're doing until things stop working. Along with SRE folks, the position may as well be akin to a firefighter. I know this first hand having been a loaded-title myself. You haven't really lived until you get paged at 2am because something is wrong with the deploys in South East Asia.

Like any software "engineering" position, there's an art and skill to determining what might be wrong, usually because you didn't build the shit yourself. That's okay though. We learn from experience and move on. Or, if you're reading my blog, you learn from me and my experience, and I receive warm-fuzzies in exchange. So, if I'm in a position where I have to create a Clojure CI/CD pipeline, as I often am, here's what I usually do.

Continuous Irritation (CI)

I'm not certain, but I think Continuous Integration means continuous build pipeline minutes billed to clients. Most clients use everybody's favourite version control: Github. When they're not trying to destroy the Software developer profession, Github serves as managed Git repositories. In the realm of professional software development, services like Github and Gitlab (and BitBucket, if you're a corpo) have completely supplanted managing Git repositories by hand. It is where our code lives, but you already know that.

Github comes with a dandy little package management solution as well. You just didn't look hard enough because it's covered in dust. Seriously, I haven't met anyone who uses the Github Packages private maven repository except for me. So lets go ahead and set that up.

Setting up Github Packages for private use is remarkably similar to setting up an AWS S3 bucket as our private repository:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">

  <servers>
    <server>
      <id>github</id>
      <username>GITHUB_USERNAME_HERE</username>
      <password>PERSONAL_ACCESS_TOKEN_HERE</password>
    </server>
  </servers>

</settings>

Put this in your ~/.m2/settings.xml file. You'll notice it requires a Github Personal Access Token (with Github Packages Read & Write permissions). This will give maven the credentials to deploy to Github Packages. Depending on your Github subscription, this can cost money.

Now we have to come up with a method to deploy if we want to deploy. I think using a tools.build script fits the bill perfectly. Let's go ahead and create a build alias in deps.edn:

...
:aliases {:build {:deps {slipset/deps-deploy {:mvn/version "0.2.0"}
                          io.github.clojure/tools.build {:mvn/version "0.9.6"}}
                   :ns-default build}}
...

We're going to use deps-deploy to help us deploy our code, but we'll get to the deploying in a bit. First let's focus on building the thing with our build.clj:

(ns build
  (:refer-clojure :exclude [test])
  (:require [clojure.tools.build.api :as b]
            [deps-deploy.deps-deploy :as dd]))

(def lib 'com.janetacarr/clojure-for-pros)
(def version (format "0.1.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file (format "target/%s-%s-standalone.jar" (name lib) version))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src" "public"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :ns-compile '[clojure-for-pros.core]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'clojure-for-pros.core}))

Our build.clj copies our source code to our working directory, class-dir, with the copy-dir function. Then, compile-clj compiles our Clojure source files as described in the basis to the working directory. The basis, or runtime basis, is a configuration map that contains the source directories from deps.edn as well as runtime information like dependencies. Then it's also used to bundle the working directory into an uberjar with b/user. Now we can create a the uberjar with the command clj -T:build uber from the project directory, and run the jar with java -jar target/clojure-for-pros-standalone.jar.

Continuous Distraction (CD)

So far, this is straight out of the tools.build Guide (mostly). What we really need to do is write a deploy function. Simple enough thanks to deps-deploy:

(ns build
  (:refer-clojure :exclude [test])
  (:require [clojure.tools.build.api :as b]))

(def lib 'com.janetacarr/clojure-for-pros)
(def version (format "0.1.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file (format "target/%s-standalone.jar" (name lib)))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src" "public"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :ns-compile '[clojure-for-pros.core]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'clojure-for-pros.core}))

(defn deploy [_]
  (let [dd-deploy (try (requiring-resolve 'deps-deploy.deps-deploy/deploy) (catch Throwable _))]
    (if dd-deploy
      (dd-deploy {:installer :remote
                  :artifact (b/resolve-path uber-file)
                  :repository "github"
                  :pom-file (b/pom-path {:lib lib 
                                         :class-dir class-dir})})
      (println "borked"))))

Running clj -T:build deploy comes back with an interesting error:

:repository has value github which is an unknown :mvn/repos
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
/Users/janet/clojure-for-pros/target/classes/META-INF/maven/com.janetacarr/clojure-for-pros/pom.xml (No such file or directory)

Full report at:
/var/folders/py/pc9dw_557nl93lsc4npft5j00000gn/T/clojure-6437169465784041420.edn

What the heck is a pom.xml file? A POM file, as it's called, is an XML file that contains data Maven uses to build a project. It also happens to be where our Maven repository data will live. A Maven repository needs an artifact(jar) POM file to determine the coordinates of the artifact. We could write a pom.xml file for our project and write its contents to our working directory. In fact, tools.build has a function expressly for this purpose called write-pom. So let's go ahead and do that.

;; updated uber function from clojure-for-pros
;; build.clj file
(defn uber [_]
  (clean nil)
  ;; default pom file path is "./pom.xml"
  (b/write-pom {:class-dir class-dir
                :lib lib
                :version version
                :basis basis
                :src-dirs ["src"]}) 
  (b/copy-dir {:src-dirs ["src" "public"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :ns-compile '[clojure-for-pros.core]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'clojure-for-pros.core}))

And we need a pom.xml file too. Most of the final POM in our uberjar will be generated by the run-time basis, but we need this bit for our private maven repo:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <distributionManagement>
    <repository>
      <id>github</id>
      <name>janetacarr github packages</name>
      <url>https://maven.pkg.github.com/janetacarr/clojure-for-pros</url>
    </repository>
  </distributionManagement>
</project>

We also need to add a :mvn/repos key-value pair to the top-level of deps.edn. This will take care of the ":repository has value github which is an unknown :mvn/repos" error message.

:mvn/repos {"github" {:url "https://maven.pkg.github.com/janetacarr/clojure-for-pros"}}

Finally, we can deploy with clj -T:build deploy.

$ clj -T:build deploy
Deploying com.janetacarr/clojure-for-pros-0.1.11 to repository github as janetacarr
Sending com/janetacarr/clojure-for-pros/0.1.11/clojure-for-pros-0.1.11.pom (7k)
    to https://maven.pkg.github.com/janetacarr/clojure-for-pros/
Sending com/janetacarr/clojure-for-pros/0.1.11/clojure-for-pros-0.1.11-standalone.jar (671195k)
    to https://maven.pkg.github.com/janetacarr/clojure-for-pros/
Retrieving com/janetacarr/clojure-for-pros/maven-metadata.xml (1k)
    from https://maven.pkg.github.com/janetacarr/clojure-for-pros/
Sending com/janetacarr/clojure-for-pros/maven-metadata.xml (1k)
    to https://maven.pkg.github.com/janetacarr/clojure-for-pros/

Trust me, bro

Since this is all done in a private repository, it feels very "trust me, bro", so I'm going to do similar things for an open-source library on my Github called quadtree-cljc. quadtree-cljc is just a library for building quadtrees and finding out which nodes intersect.

(ns build
  (:require [clojure.tools.build.api :as b]))

(def lib 'com.janetacarr/quadtree-cljc)
(def version "0.1.4")
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def jar-file (format "target/%s-%s.jar" lib version))

(defn clean [_]
  (b/delete {:path "target"}))

(defn- pom-template [version]
  [[:description "A quadtree implementation for Clojure(script)"]
   [:url "https://github.com/janetacarr/quadtree-cljc"]
   [:licenses
    [:license
     [:name "APACHE LICENSE, VERSION 2.0"]
     [:url "https://www.apache.org/licenses/LICENSE-2.0"]]]
   [:developers
    [:developer
     [:name "Janet A. Carr"]]]
   [:scm
    [:url "https://github.com/janetacarr/quadtree-cljc"]
    [:connection "scm:git:https://github.com/janetacarr/quadtree-cljc.git"]
    [:developerConnection "scm:git:ssh://git@github.com/janetacarr/quadtree-cljc.git"]
    [:tag version]]])

(defn jar [_]
  (clean nil)
  (b/write-pom {:class-dir class-dir
                :lib lib
                :version version
                :basis basis
                :src-dirs ["src"]
                :pom-data (pom-template version)})
  (b/copy-dir {:src-dirs ["src"]
               :target-dir class-dir})
  (b/jar {:class-dir class-dir
          :jar-file jar-file}))

(defn deploy
  [{:keys [repository] :as opts}]
  (let [dd-deploy (try (requiring-resolve 'deps-deploy.deps-deploy/deploy) (catch Throwable _))]
    (if dd-deploy
      (dd-deploy {:installer :remote
                  :artifact (b/resolve-path jar-file)
                  :repository (or (str repository) "clojars")
                  :pom-file (b/pom-path {:lib lib
                                         :class-dir class-dir})})
      (println "borked"))))

Since quadtree-cljc is open-source, it might be a good idea to let developers choose where to upload this jar by default, this is where repository comes in. If we want to deploy to Github, we just use the command clj -T:build deploy :repository github provided we have both the pom.xml anddeps.edn :mvn/repos set up correctly for quadtree-cljc like above. Because I also deploy quadtree-cljc to Clojars, I opted for pom-template (with a licenses) using hiccup style syntax instead of having a pom.xml.

Actions speak louder than words

Creating JARs and putting them into Maven repos isn't really a CI/CD pipeline now is it? Ideally, we'd have a Github Organization account, and use Github packages to store library JARs and our app Docker Images, gluing it all together with Github Actions or CircleCI (or Jenkins, corpo). I don't want to pay for Github Organizations, so let's go ahead and set up some Github Actions on regular Github.

For quadtree-cljc we'll set up a Github Actions to build our jar and deploy it to clojars whenever a pull request (PR) is merged into master. It's also probably a good idea that the Action trigger, test, and build the PR code before a merge. You can find the Github Action YAML here.

We'll do something similar for clojure-for-pros. Instead of just deploying the uberjar to Github's private Maven repo, we're going to build a Docker image that will run the uberjar, and then trigger a deploy to production with the Docker image.

name: Clojure Uberjar and Docker CI/CD

on:
  pull_request:
    branches: [ master, main ]
  push:
    branches: [ master, main ]

jobs:
  build-and-push-docker:
    runs-on: ubuntu-latest
    needs: build-and-test-uberjar
    if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main')
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 20
        uses: actions/setup-java@v2
        with:
          java-version: '20'
          distribution: 'temurin'
      - name: Install Clojure
        run: |
          curl -L -O https://github.com/clojure/brew-install/releases/latest/download/posix-install.sh
          chmod +x posix-install.sh
          sudo ./posix-install.sh
      - name: Build Uberjar
        run: clojure -T:build uber
      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GH_PAT }}
      - name: Build and push Docker image
        run: |
          docker build --platform=linux/amd64 -t ghcr.io/${{ github.repository }}/clojure-for-pros:${{ github.sha }} .
          docker push ghcr.io/${{ github.actor }}/clojure-for-pros:${{ github.sha }}
          docker push ghcr.io/${{ github.actor }}/clojure-for-pros:latest

FROM clojure:temurin-20-alpine
COPY . /app
WORKDIR /app

ENV DATABASE_URL=jdbc:postgresql://localhost:5432/postgres?user=postgres\&password=postgres
ENV PORT=8080
ENV API_HOST=localhost:8080
ENV CLOUDFRONT_KEY=""
ENV TOKEN_SECRET=""
ENV DEV_MODE=false

CMD java -jar target/clojure-for-pros.jar

Party Time

Once all that is done, we can trigger a deploy from a Github Action. I primarily deploy on Render. It's like a Heroku competitor. Typically, If I'm lazy (I am), I'll just build the docker image inside the Render build pipeline and be done with it.

But, this is a CI/CD pipeline, damn it! We're going to do the real thing™️ and set up deploys to production. Here's a job to hit Render's deploy hook when the Docker image is built.

....
  notify-render:
    runs-on: ubuntu-latest
    needs: build-and-push-docker
    if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main')
    steps:
      - name: Notify Render.com
        run: |
          curl https://api.render.com/deploy/srv-fakefakefakefake?key=${{ secrets.RENDER_API_TOKEN }}

And there you have it. A Clojure CI/CD pipeline. With the right credentials, builds can pull dependencies and Docker images from Github Packages.

Conclusion

None of this is very trivial to understand. Most of it is buried in obscure blog posts, documentation, and codebases, but is a core piece of a Software Architecture, so I felt it deserved along blog post of it's own.

I do want to give a huge shout out to Sean Corfield for his now deprecated build-clj helper library. Between that and the tools.build documentation, I was able to work through this.