Devops is one of those important positions where nobody cares what you're doing until things stop working. Along with SRE folks, the position may as well be akin to a firefighter. I know this first hand having been a loaded-title myself. You haven't really lived until you get paged at 2am because something is wrong with the deploys in South East Asia.

Like any software "engineering" position, there's an art and skill to determining what might be wrong, usually because you didn't build the shit yourself. That's okay though. We learn from experience and move on. Or, if you're reading my blog, you learn from me and my experience, and I receive warm-fuzzies in exchange. So, if I'm in a position where I have to create a Clojure CI/CD pipeline, as I often am, here's what I usually do.

Continuous Irritation (CI)

I'm not certain, but I think Continuous Integration means continuous build pipeline minutes billed to clients. Most clients use everybody's favourite version control: Github. When they're not trying to destroy the Software developer profession, Github serves as managed Git repositories. In the realm of professional software development, services like Github and Gitlab (and BitBucket, if you're a corpo) have completely supplanted managing Git repositories by hand. It is where our code lives, but you already know that.

Github comes with a dandy little package management solution as well. You just didn't look hard enough because it's covered in dust. Seriously, I haven't met anyone who uses the Github Packages private maven repository except for me. So lets go ahead and set that up.

Setting up Github Packages for private use is remarkably similar to setting up an AWS S3 bucket as our private repository:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">

  <servers>
    <server>
      <id>github</id>
      <username>GITHUB_USERNAME_HERE</username>
      <password>PERSONAL_ACCESS_TOKEN_HERE</password>
    </server>
  </servers>

</settings>

Put this in your ~/.m2/settings.xml file. You'll notice it requires a Github Personal Access Token (with Github Packages Read & Write permissions). This will give maven the credentials to deploy to Github Packages. Depending on your Github subscription, this can cost money.

Now we have to come up with a method to deploy if we want to deploy. I think using a tools.build script fits the bill perfectly. Let's go ahead and create a build alias in deps.edn:

...
:aliases {:build {:deps {slipset/deps-deploy {:mvn/version "0.2.0"}
                          io.github.clojure/tools.build {:mvn/version "0.9.6"}}
                   :ns-default build}}
...

We're going to use deps-deploy to help us deploy our code, but we'll get to the deploying in a bit. First let's focus on building the thing with our build.clj:

(ns build
  (:refer-clojure :exclude [test])
  (:require [clojure.tools.build.api :as b]
            [deps-deploy.deps-deploy :as dd]))

(def lib 'com.janetacarr/clojure-for-pros)
(def version (format "0.1.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file (format "target/%s-%s-standalone.jar" (name lib) version))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src" "public"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :ns-compile '[clojure-for-pros.core]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'clojure-for-pros.core}))

Our build.clj copies our source code to our working directory, class-dir, with the copy-dir function. Then, compile-clj compiles our Clojure source files as described in the basis to the working directory. The basis, or runtime basis, is a configuration map that contains the source directories from deps.edn as well as runtime information like dependencies. Then it's also used to bundle the working directory into an uberjar with b/user. Now we can create a the uberjar with the command clj -T:build uber from the project directory, and run the jar with java -jar target/clojure-for-pros-standalone.jar.

Continuous Distraction (CD)

So far, this is straight out of the tools.build Guide (mostly). What we really need to do is write a deploy function. Simple enough thanks to deps-deploy:

(ns build
  (:refer-clojure :exclude [test])
  (:require [clojure.tools.build.api :as b]))

(def lib 'com.janetacarr/clojure-for-pros)
(def version (format "0.1.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file (format "target/%s-standalone.jar" (name lib)))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src" "public"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :ns-compile '[clojure-for-pros.core]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'clojure-for-pros.core}))

(defn deploy [_]
  (let [dd-deploy (try (requiring-resolve 'deps-deploy.deps-deploy/deploy) (catch Throwable _))]
    (if dd-deploy
      (dd-deploy {:installer :remote
                  :artifact (b/resolve-path uber-file)
                  :repository "github"
                  :pom-file (b/pom-path {:lib lib 
                                         :class-dir class-dir})})
      (println "borked"))))

Running clj -T:build deploy comes back with an interesting error:

:repository has value github which is an unknown :mvn/repos
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
/Users/janet/clojure-for-pros/target/classes/META-INF/maven/com.janetacarr/clojure-for-pros/pom.xml (No such file or directory)

Full report at:
/var/folders/py/pc9dw_557nl93lsc4npft5j00000gn/T/clojure-6437169465784041420.edn

What the heck is a pom.xml file? A POM file, as it's called, is an XML file that contains data Maven uses to build a project. It also happens to be where our Maven repository data will live. A Maven repository needs an artifact(jar) POM file to determine the coordinates of the artifact. We could write a pom.xml file for our project and write its contents to our working directory. In fact, tools.build has a function expressly for this purpose called write-pom. So let's go ahead and do that.

;; updated uber function from clojure-for-pros
;; build.clj file
(defn uber [_]
  (clean nil)
  ;; default pom file path is "./pom.xml"
  (b/write-pom {:class-dir class-dir
                :lib lib
                :version version
                :basis basis
                :src-dirs ["src"]}) 
  (b/copy-dir {:src-dirs ["src" "public"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :ns-compile '[clojure-for-pros.core]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'clojure-for-pros.core}))

And we need a pom.xml file too. Most of the final POM in our uberjar will be generated by the run-time basis, but we need this bit for our private maven repo:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <distributionManagement>
    <repository>
      <id>github</id>
      <name>janetacarr github packages</name>
      <url>https://maven.pkg.github.com/janetacarr/clojure-for-pros</url>
    </repository>
  </distributionManagement>
</project>

We also need to add a :mvn/repos key-value pair to the top-level of deps.edn. This will take care of the ":repository has value github which is an unknown :mvn/repos" error message.

:mvn/repos {"github" {:url "https://maven.pkg.github.com/janetacarr/clojure-for-pros"}}

Finally, we can deploy with clj -T:build deploy.

$ clj -T:build deploy
Deploying com.janetacarr/clojure-for-pros-0.1.11 to repository github as janetacarr
Sending com/janetacarr/clojure-for-pros/0.1.11/clojure-for-pros-0.1.11.pom (7k)
    to https://maven.pkg.github.com/janetacarr/clojure-for-pros/
Sending com/janetacarr/clojure-for-pros/0.1.11/clojure-for-pros-0.1.11-standalone.jar (671195k)
    to https://maven.pkg.github.com/janetacarr/clojure-for-pros/
Retrieving com/janetacarr/clojure-for-pros/maven-metadata.xml (1k)
    from https://maven.pkg.github.com/janetacarr/clojure-for-pros/
Sending com/janetacarr/clojure-for-pros/maven-metadata.xml (1k)
    to https://maven.pkg.github.com/janetacarr/clojure-for-pros/

Trust me, bro

Since this is all done in a private repository, it feels very "trust me, bro", so I'm going to do similar things for an open-source library on my Github called quadtree-cljc. quadtree-cljc is just a library for building quadtrees and finding out which nodes intersect.

(ns build
  (:require [clojure.tools.build.api :as b]))

(def lib 'com.janetacarr/quadtree-cljc)
(def version "0.1.4")
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def jar-file (format "target/%s-%s.jar" lib version))

(defn clean [_]
  (b/delete {:path "target"}))

(defn- pom-template [version]
  [[:description "A quadtree implementation for Clojure(script)"]
   [:url "https://github.com/janetacarr/quadtree-cljc"]
   [:licenses
    [:license
     [:name "APACHE LICENSE, VERSION 2.0"]
     [:url "https://www.apache.org/licenses/LICENSE-2.0"]]]
   [:developers
    [:developer
     [:name "Janet A. Carr"]]]
   [:scm
    [:url "https://github.com/janetacarr/quadtree-cljc"]
    [:connection "scm:git:https://github.com/janetacarr/quadtree-cljc.git"]
    [:developerConnection "scm:git:ssh://git@github.com/janetacarr/quadtree-cljc.git"]
    [:tag version]]])

(defn jar [_]
  (clean nil)
  (b/write-pom {:class-dir class-dir
                :lib lib
                :version version
                :basis basis
                :src-dirs ["src"]
                :pom-data (pom-template version)})
  (b/copy-dir {:src-dirs ["src"]
               :target-dir class-dir})
  (b/jar {:class-dir class-dir
          :jar-file jar-file}))

(defn deploy
  [{:keys [repository] :as opts}]
  (let [dd-deploy (try (requiring-resolve 'deps-deploy.deps-deploy/deploy) (catch Throwable _))]
    (if dd-deploy
      (dd-deploy {:installer :remote
                  :artifact (b/resolve-path jar-file)
                  :repository (or (str repository) "clojars")
                  :pom-file (b/pom-path {:lib lib
                                         :class-dir class-dir})})
      (println "borked"))))

Since quadtree-cljc is open-source, it might be a good idea to let developers choose where to upload this jar by default, this is where repository comes in. If we want to deploy to Github, we just use the command clj -T:build deploy :repository github provided we have both the pom.xml anddeps.edn :mvn/repos set up correctly for quadtree-cljc like above. Because I also deploy quadtree-cljc to Clojars, I opted for pom-template (with a licenses) using hiccup style syntax instead of having a pom.xml.

Actions speak louder than words

Creating JARs and putting them into Maven repos isn't really a CI/CD pipeline now is it? Ideally, we'd have a Github Organization account, and use Github packages to store library JARs and our app Docker Images, gluing it all together with Github Actions or CircleCI (or Jenkins, corpo). I don't want to pay for Github Organizations, so let's go ahead and set up some Github Actions on regular Github.

For quadtree-cljc we'll set up a Github Actions to build our jar and deploy it to clojars whenever a pull request (PR) is merged into master. It's also probably a good idea that the Action trigger, test, and build the PR code before a merge. You can find the Github Action YAML here.

We'll do something similar for clojure-for-pros. Instead of just deploying the uberjar to Github's private Maven repo, we're going to build a Docker image that will run the uberjar, and then trigger a deploy to production with the Docker image.

name: Clojure Uberjar and Docker CI/CD

on:
  pull_request:
    branches: [ master, main ]
  push:
    branches: [ master, main ]

jobs:
  build-and-push-docker:
    runs-on: ubuntu-latest
    needs: build-and-test-uberjar
    if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main')
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 20
        uses: actions/setup-java@v2
        with:
          java-version: '20'
          distribution: 'temurin'
      - name: Install Clojure
        run: |
          curl -L -O https://github.com/clojure/brew-install/releases/latest/download/posix-install.sh
          chmod +x posix-install.sh
          sudo ./posix-install.sh
      - name: Build Uberjar
        run: clojure -T:build uber
      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GH_PAT }}
      - name: Build and push Docker image
        run: |
          docker build --platform=linux/amd64 -t ghcr.io/${{ github.repository }}/clojure-for-pros:${{ github.sha }} .
          docker push ghcr.io/${{ github.actor }}/clojure-for-pros:${{ github.sha }}
          docker push ghcr.io/${{ github.actor }}/clojure-for-pros:latest

FROM clojure:temurin-20-alpine
COPY . /app
WORKDIR /app

ENV DATABASE_URL=jdbc:postgresql://localhost:5432/postgres?user=postgres\&password=postgres
ENV PORT=8080
ENV API_HOST=localhost:8080
ENV CLOUDFRONT_KEY=""
ENV TOKEN_SECRET=""
ENV DEV_MODE=false

CMD java -jar target/clojure-for-pros.jar

Party Time

Once all that is done, we can trigger a deploy from a Github Action. I primarily deploy on Render. It's like a Heroku competitor. Typically, If I'm lazy (I am), I'll just build the docker image inside the Render build pipeline and be done with it.

But, this is a CI/CD pipeline, damn it! We're going to do the real thing™️ and set up deploys to production. Here's a job to hit Render's deploy hook when the Docker image is built.

....
  notify-render:
    runs-on: ubuntu-latest
    needs: build-and-push-docker
    if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main')
    steps:
      - name: Notify Render.com
        run: |
          curl https://api.render.com/deploy/srv-fakefakefakefake?key=${{ secrets.RENDER_API_TOKEN }}

And there you have it. A Clojure CI/CD pipeline. With the right credentials, builds can pull dependencies and Docker images from Github Packages.

Conclusion

None of this is very trivial to understand. Most of it is buried in obscure blog posts, documentation, and codebases, but is a core piece of a Software Architecture, so I felt it deserved along blog post of it's own.

I do want to give a huge shout out to Sean Corfield for his now deprecated build-clj helper library. Between that and the tools.build documentation, I was able to work through this.

People often come to me asking "I love the idea of Clojure, but how do you write code without types?". I struggle to answer this question. I have no idea what they're talking about half the time. The nuance of strong and weak typing, and static vs dynamic type checking, is often lost on people. I think they want their tools to tell them what to do. I don't struggle with that like others. I never have. It never occured to me that I might just be a weirdo.

Weird life

Like a lot of things, I think I can directly attribute my cognitive model of writing code to how I learned to program. Right around the time I was getting into Linux in high-school and becoming a script kiddy. I learned how to program in C. At first on Windows with something called Bloodshed Dev C++, then I quickly graduated to the terminal because of this cool thing called Linux. I used GCC and GEdit for a long time. Not because I'm some arogant asshole (debateable). I didn't really know better. Then my hacky brain stole this book called Hacking: The Art of Exploitation. At sixteen I learned how to disassemble the C programs I had written and attach GNU debugger (GDB) to examine memory, graduating from:

printf("HERE DUMBASS: %s", *buf);

This "bad" habit of GEdit + command line followed me to University where I coasted and watched people new to programming struggle writing Java. I used it during labs, competitions, assignments, etc. The first time I was actually introduced to Eclipse was when my friend used it on our Intro to Software Engineering project.

All of this was made worse when I finally dropped out of my Computer Science program to work in the industry. My first real developer job was writing Clojure, and the office preference was to use Emacs prelude + CIDER.

Every office has it problems, but what I quickly learned was that documentation was always out of date. I had to build the muscle memory to just read the code which wasn't out of date, ever. I suspect this is why I always find myself reading the Clojure source code for interesting technical nuggets. Even long after I was a Go developer, I still read code everywhere.

Okay, so what?

The less code kept in the developer's head the better. The typical model of software development follows a loop of getting product requirements, understanding enough of the codebase to formulate a solution, and then implementing that solution. More code in the head leads to more complexity.

Blackboxing code addresses complexity by reducing something strictly to inputs and outputs. I think people coming from statically type-checked languages conflate complier errors and type declarations with blackboxing. In reality, they've traded flexibility for guardrails. While I argee that dynamic type checking requires a little bit more cognitive load, docstrings and idioms cut down on cognitive load, and they are far more descriptive than type declarations on function arguments.

Similarly, by sending type mismatching off to the compile time, rather than runtime, those developers lose out on the flexibility on deciding what to do with that type mismatch. Of course, the type mismatch fault can mitigated by testing and assertions. Often, it doesn't matter in Clojure anyways.

While Clojure is a dynamically type-checked language, it is also strongly-typed. Clojure types are several types at once rather than implicitly converting types between function calls. Collections (vectors, maps, sets, and lists) adhere to enough type interfaces they work with many functions like filter, reduce, partition, group-by, etc. get, assoc, and update not only operate on maps, but can also operate on vectors and so on.

However, while developers might have a hard time shooting themselves in the foot, that doesn't mean they can't shoot their colleagues in the foot. Here's some ways to avoid that:

Idioms

Clojure idioms convey information about the type of value being passed a function, macro, and methods at a glance like type declarations. They are great for smaller, general functions.

Follow clojure.core's example for idiomatic names like pred and coll.
in functions:

f, g, h - function input

n - integer input usually a size

index, i - integer index

x, y - numbers

xs - sequence

m - map

k, ks - key, keys

(defn drop-last
  "Return a lazy sequence of all but the last n (default 1) items in coll"
  {:added "1.0"
   :static true}
  ([coll] (drop-last 1 coll))
  ([n coll] (map (fn [x _] x) coll (drop n coll))))

Idioms only capture part of the story, and are not entirely suitable for all cases. Sometimes, we want our bindings or argument names to convey more domain information. We can describe them in docstrings.

Docstrings

Docstrings are first-class documentation for functions. Clojure tools like editors and documentation generators look at docstrings first, so It's important to do well. Docstrings should describe what the function does, takes as parameters, and returns. Use backticks functions parameters and other important bits like so:

(defn insert
  "Insert `bounds-obj` into the node, returning a freshly grown quadtree.
  If the node exceeds the capacity, it will split and add all objects to
  their corresponding subnodes."
  [quadtree bounds-obj]
  (let [{:keys [nodes objects bounds
                level max-levels
                max-objects objects]} quadtree
        all-objects (conj objects bounds-obj)]
    (if (pos? (count nodes))
      (as-> quadtree quadtree
        (assoc quadtree :objects [])
        (reduce (fn [quadtree obj]
                  (let [quadrant (get-quadrant quadtree obj)
                        nodes (:nodes quadtree)]
                    (if quadrant
                      (merge quadtree {:nodes (assoc nodes
                                                     quadrant
                                                     (insert (nth nodes quadrant)
                                                             obj))})
                      (update quadtree :objects #(conj % obj)))))
                quadtree
                all-objects))
      (if (and (> (count all-objects) max-objects) (< level max-levels))
        (let [quadtree (if (empty? nodes) (split quadtree) quadtree)]
          (insert quadtree bounds-obj))
        (merge quadtree {:objects all-objects})))))

Comments

Similar to docstrings are Clojure's comments. These are comments like any other comments from any language, but Clojure comes with a couple more types of comments:

• semi-colon comments ; - typical boring comments, completely ignored
• reader comments #_ - tells the reader to ignore the next form
• rich comments (comment ....) - form evaluates contents, but unreachable

;; This comment is ignored by the complier
;; It's great for long form comments. 
;; and some editors allow storing and evaluating
;; REPL-like expressions here.

#_(defn hello [] (println "This form ignored, and not evaluated"))

(comment
  (def hello "This comment IS evaluated, but not reachable by code))

REPL

Some IDEs allow storing and evaluating REPL-like expressions in Clojure. If you haven't heard of a REPL, it's a Read-Eval-Print-Loop. Clojure features a robust REPL compared to other programming languages as developers can spin up their application process and access the state of the application while it's running.

It's great for debugging and hardening code, and the closest to thing to static type checked process of compiling and waiting for type errors. Except the Clojure REPL doesn't have to build the entire program again, so it can be much faster to iterate with than a statically type-checeked compiler.

Here's a dev script I use to bootstrap my process with the main entry point :main-opts ["dev.clj"] in my deps.edn alias. I use (CIDER) nREPL to connect to the process once it's running.

(defmulti task first)

(defmethod task :default
  [[task-name]]
  (println "Unknown task:" task-name)
  (System/exit 1))

(defmethod task nil
  [_]
  (require '[shadow.cljs.devtools.cli :as shadow])
  (require '[app.core :as app])
  ((resolve 'app/-main))
  ((resolve 'shadow/-main) "watch" "app"))

(defmethod task "repl"
  [_]
  (clojure.main/repl :init #(doto 'app.core require in-ns)))

(task *command-line-args*)

Once connected, I can deref the server atom or alter a root var to enable debug logs or prototype functions for my current dev task. nREPL can be very handy to debug production systems as it can be used with a SSH tunnel for secure access.

Clojure.spec

My thoughts on spec are well known at this point. I really don't like bolting on quasi-static-but-at-run-time-type-checking. I find it's best used for the system boundaries to check and coerce input as well as some critical code pathways. Developers should exercise caution when using spec though. Codebases with too much spec suffer from rigidity, making changes harder.

Rigidity is brought on by the false promise of spec test generators (using spec to generate function test cases). If you want to use generators, you can't just sprinkle in some Clojure spec. Spec generators are great at generating obscure test cases. They create a cycle of tweaking functions and specs, so it doesn't generate unintended non-sense and the function can handle the desired input. I find the time spent on this has little value for the payoff and makes codebases more difficult to develop in.

If you've ever wanted to build the event-processing startup of your dreams and exit for billions of dollars, this post is for you, or you just want to know how to code one. I didn't know how they worked at my first developer job. We had this events pipeline with all the frills: A highly-available event bus, definitions, consumers, producers, consumer-producers. I always figured the architecture just started like that. Right out of the gate, we're scalling for millions of events per second. Now that I fancy myself as a lean startup person (or whatever), I think a lot in terms of Minimum-Viable-Product (MVP).

You can use this architecture as your blueprint for any event handling service. I think the ideas here will translate well enough. If you're new to event-driven architecture, we decouple receiving events from processing events with one hell-of-a-buffer called an event bus. Consumers consume messages from the event bus, and producers produce messages by putting them on the event bus. We segregate the events by attributes or shape with some kind of publish/subscribe technique (or message broker).

Lucky for us, Clojure has two types of publish/subscribe facilities built-in. The first is a watch, quite literally a first-class observer. The second, channels, and the one we'll be using, is built into clojure's core.async library. It allows us to supply a function to create our publication topic. Let's use this for a stand-in until we hit millions of events and exit for billions with our new event-processing startup.

Enter the Webhooks-as-a-Service

Event driven architecture MVP is relatively straight forward.

If you're a long time reader, you know where this is heading. We're going to sandwich our event-bus in between a few higher-order functions. Not only can we decouple the event-bus from the application and swap it out for something more 'cloud', we can also supply a multimethod and dispatch on the incoming source or outgoing destination. Dispatching like that on a multimethod gives us the potential for having multiple event buses. Like a poor man's sharding, or in reality a way to deprioritize certain events. All from passing a function as an argrument!

Okay, real simple, lets create an event bus with a core.async channel, and we'll spin up our event bus and message broker.

(defn payment-received-consumer
  [events-chan]
  (a/go-loop [event (a/<! events-chan)]
    ;; Do something with event
    (println "Received payment" event)))

(defn user-unsubscribed-consumer
  [events-chan]
  (a/go-loop [event (a/<! events-chan)]
    ;; Do something with event
    (println "User unsubscribed" event)))

(defn ->event-deserializer
  [publish-fn]
  (fn event-deserializer
    [event-chan]
    (a/go-loop [event (a/<! event-chan)]
      (try
        (let [parsed-event (json/parse-string event)]
          (if (valid-event? parsed-event)
            (publish-fn parsed-event)
            (log/warn "Dropping unsigned event: " event)))
        (catch Exception e
          (log/error "Internal error processing event: "
                     event
                     (.getMessage e))))
      (recur (a/<! event-chan)))))

(defn setup-consumer
  [consumer-fn publication topic]
  (let [consumer-chan (a/chan)]
    (a/sub publication topic consumer-chan)
    (consumer-fn consumer-chan)))

(def event-bus (a/chan 2048))
(def publication (a/pub event-bus :event-type))
(def payments-consumer (setup-consumer payment-received-consumer
                                       publication
                                       :payment))
(def user-consumer (setup-consumer user-unsubscribed-consumer
                                   publication
                                   :unsubscribed))
                                   
(def serializing-consumer-producer (setup-consumer (->event-deserializer publish-event)
                                                   publication
                                                   nil))

core.async/pub creates a 'publication' on channel event-bus and a topic function, :event-type (taking advantage of the iFn interface). For now we'll assume the shape of our events are all the same, and we'll only receive events that tell us what they are via the :event-type keyword.

We also have a topic nil. We'll use this topic for a consumer-producer that de-serializes JSON to EDN for us. If you're wondering why on earth we'd want to do that, most streaming tech (eg AWS kinesis) passes JSON strings through to our consumers. We could also serialize JSON to EDN-encoded strings and put them back on the bus. This will also give us the ability to respond to requests faster since we don't have to process the body of the request (yet); we'll return HTTP status code 202 whenever we receive an event.

Of course, just putting raw events on the event bus, and de-serializing them back onto the event bus will create back pressure. There are techniques for dealing with back pressure I won't go into here. To manage initial back pressure, I will say we should supply a buffer to our event-bus channel. Channels created without one create an unbuffered channel where the value will only take if there is a put waiting. Without a buffer, back pressure on the default buffer of put! and take! will rise too quickly depending on our context. We can take back pressure on our buffered channel to be a good sign though: We can start burning captial on AWS infrastructure, signalling our soon-to-be billion dollar exit.

Consumers, Producers, and Consumer-Producers

You'll notice in the setup-consumer adapter contains an internal unbuffered channel, consumer-chan, passed to consumer-fn. consumer-fn exibits our consumer interface, a function that takes a channel. This makes for nice core.async programming in our consumers regardless of what the event bus looks like. If we wanted to get super fancy, we could store descriptions of the event data ("Event Definitions") in a database to further decouple our consumers from the events they handle.

On the other hand, producers are pretty simple:

(defn publish-event
  [event]
  (a/go (a/>! event-bus event)))
  
(defn ->event-handler
  [publish-fn]
  (fn [req]
    (let [{:keys [body]} req]
      (publish-fn body)
      {:status 202})))

The ->event-handler constructor creates a request handler for our system boundary, closing over publish-event. Any consumers that want to produce events, like event-deserializer, can close over publish-event, or just call it directly.

Caveats

We can't talk about event-driven anything without talking about eventual consistency. By introducing an event bus, or even an event buffer, we've decoupled receiving events from processing events. We now have a lead time between event generation, and it's intended effect on the system. Typically, it's a bad fit for certain types of real-time or responsive applications unless the lead time can be well managed into the milliseconds. A challenging feat without sufficient software engineering resources.  

Conclusion?

And that's pretty much it for our MVP. Add consumers as needed. You can find the full source code on Github. Since I was dragging my heels on this post, I decided to break it up into a two (or three) part series. In part two I'll separate out the event bus into AWS Kinesis and the consumers into AWS lambdas with Holy Lambda. Remember, billions.

This odd thing keeps happening to me where companies reach out to ask if I know of any Clojure developers they can hire instead of just hiring me. I understand why they do this. I am not interested in full-time position, after all. I think most people would bitterly look at this as if the company were asking for free labour. In some cases, I agree with this sentiment. However, when it comes to Clojure developers, I can empathize as I was something of a hiring manager once. The pipeline doesn't exist.

Do you really need a full-time dev?

You should think hard about this question. We're in the midst of a recession, so people with existing Clojure jobs or experience will probably be reluctant to jump ship from their current job since the job market has begun to dry up (for now). If you're sinking an unreasonable amount of time into searching for a full-time Clojure(script) developer, why not consider a contractor or agency? If Clojure/Conj 2023 was anything to go by, most Clojure developers are self-employed, or at least, most of the people I met are. This is compounded by the fact that most of the Clojure/Conj 2023 sponsors and tables were agencies specializing in Clojure.

Sure, there are plenty of people in the #available-for-jobs channel in on the Clojurians slack looking for full-time work right now. Because full-time positions typically can't cross nationalities (not without legal), a company's nationality rarely intersects with the people in the channel. I did a quick search for a Canadian in the channel, and there was only explicitly one Canadian in there. Companies could use a service like remote.com to hire Clojure developers around the world, but that creates two problems: They might not be a remote first company (I know, I know), or they might not want to spin up a managed subsidiary in a different country or whatever.

According to the StackOverflow developer survey, Clojure developers are some of the highest paid in the world. If you want to command the forty to sixty hours per week of a Clojure developer's time, you better be willing to pay for it. I see plenty of companies offering paltry pay because developers get to use Clojure.

One tactic I've seen in the past is to hire functional programmers from other developer backgrounds, or just hire some functional-curious developers, and train them to use Clojure. Easier said than done, but it might be worth considering if you're wasting too much time or money searching for a Clojure developer to hire. Either way there are ways to evaluate the Clojure, functional programmer, and functional-curious developer in an interview.

Functional Programming Interview

As a base, A functional programming interview should evaluate the use of higher-order functions, function composition, polymorphism, reducers, persistent data structures, idiomatic things, and testing. Huge bonus points if they know how to design software with them.

On top of that, it would be wise to see how familiar the interviewee is with the Clojure ecosystem. For example, do they know the current state-of-affairs for http router, webserver, database, frontend combos? Can they build and push jars to Github Packages or Clojars? Can they deploy a Clojure project to something? Do they know how to contribute to a Clojure library without creating a major version change? I know you may not use what's "standard", so your mileage may vary on this one.

For senior (and higher) level Clojure developers, You may want to evaluate whether candidates understand performance characteristics like transients, transducers, or inlining, and at this level you're probably looking for more application of Software Engineering principles. For example, you could ask questions like "By which mechanisms can you decouple parts of Clojure code with?", "How would you architect an event-driven architecture with only Clojure (no cloud/IaaS)?" or just ask them to do the take home project and look for design decisions.

A full-stack re-frame frontend and ring backend with XTDB or PostgreSQL might be a good jumping off point. An instagram clone, for example. What's the schema look like? What concerns are being separated, and how is the interviewee choosing to separate them? How did they choose to structure their re-frame events, co-effects, and effects namespaces? Should this regular function call actually be a higher-order function passed as an argument? The list goes on.

Unfortunately, there's a class of people I've included that might have higher failure rates: the functional-curious. Pick these battles on a case-by-case basis. Often less experienced folks will do Clojure faux pas without knowing. For example, using the for macro for iteration, or worse, side-effects. Other Clojure faux pas' to look out for are: using macros when not needed; state when not needed; not using core.async when doing concurrency; or not understanding the difference between atoms, agents, and refs.

Finally the best tip I can give is to see if interviewees read Clojure.core source code for a better understanding of how things work under the hood. One can gain a lot of understanding from this, half of what I know about Clojure has come from reading the Clojure.core source code on Github.

Conclusion

I think companies pursue full-time employees without really thinking about what they want out of employment. Usually, companies want full-time employees because of warped notions of work and management. I think companies hiring Clojure developers should at least consider a freelancer or agency (at the risk of sounding like an ad for my own consulting services). The 'hiring pipeline' doesn't really exist for Clojure developers. The companies that seem to be actually executing on their vision are not afraid to spend the scratch on good freelancers or an agency. At the end of the day, employment is a service-based role (developers), and results are results. There are no results if you're wasting all your time trying to hire people who aren't there.

When I started in this industry, the architecture 'all the rage' was microservices. Clojure lends itself nicely to microservices. Some developers struggle in large Clojure codebases, so the small codebases of microservices provide great refuge for the salaried functional programmer. However, the many small codebases create engineering complexity through multiple teams and services. That complexity comes at a hefty cost for software engineering budgets, so microservices aren't exactly a great approach if you have three users.

I've done the startup thing before too. After leaving the microservices place, I got a Clojure gig at a place with a Service-Oriented Architecture (SOA). Great company, but developing with their codebase felt like nails on a chalkboard because they had a monorepo with stuff all over the place. Unsurprisingly, these are the kind of companies I suspect want to move away from functional programming languages once they're no longer a fun toy to play with.

When they started looking at other options, I learned about Hexagonal Architecture also known as Ports and Adapters Architecture. Ports and Adapters Architecture decouples system inputs, business logic, and system outputs by separating them into distinct domains. If that sounds like a three-tiered architecture to you, you're not too far off, except the 'tiers' would be further broken down by domain into 'sub-tiers'. Specifically, our sub-tiers can be thought of as the adapters in this metaphor. Adapters literally meaning adapter(ish) pattern. Adapters reify the interface of a port for the benefit of the of the system inputs or outputs. Traditionally, ports are interfaces. So in Clojure speak, they could be protocols.

P&A

If you see a protocol and wince, I don't blame you. Protocols can introduce rigidity to your design, and they're tough to design like any bag of function signatures. They remind functional programmers of being in the trenches of the brittle old Object-Oriented Design. But what if we could deconstruct a protocol? If it really is just a bag of function signatures, we'd be able to decouple our domains by using functions as arguments.

;; Driven adapter for Sentry satistfies the "interface"
;; or port that `process-string` expects. In this case the
;; signature for logging-fn is:
;; (Throwable) => nil
;;
;; `capture-exception-adapter` function is adapting
;; the "interface" provided by the sentry-clj lib.
(defn capture-exception-adapter
  [exception]
  (-> exception
      (Throwable->map)
      (sentry/send-event)))

;; Core business logic expects the `logging-fn`
;; interface/port, presumably for side-effects,
;; of the function signature (Throwable) => nil
(defn process-string
  [logging-fn input]
  (try
    (apply str (reverse input))
    (catch Exception e
      (logging-fn e)
      (throw e))))

;; Our business logic Driving adapter. Here it
;; composes our business logic with the driven
;; adapter, but that doesn't necessarily need to be
;; the case. The important thing is to decouple
;; the business logic from the input(s).
(defn process-string-adapter
  [input]
  (process-string capture-exception-adapter input))

;; API handler expects an "interface" or port
;; `biz-logic-fn` that takes a string and
;; returns a string.
(defn ->handler
  [biz-logic-fn]
  (fn handler-fn
    [request]
    (let [input (get-in request [:params :input])]
      {:status 200
       :headers {"Content-Type" "application/json"}
       :body (str "{\"result\": \"" (biz-logic-fn input) "\"}")})))

;; Here at the system boundary with either inject or close
;; over our adapters
(let [handler (->handler process-string-adapter)]
  (defroutes app-routes
    (GET "/process/:input" request (handler request))
    (route/not-found "Not Found")))

(def app
  (wrap-defaults app-routes site-defaults))

In the above example, our handler factory ->handler closes over a function called biz-logic-fn.  biz-logic-fn gives us the smallest example of a port: a function that takes a string input and returns a string. The actual business logic may neither take a string, nor return a string, so we must convert our "interface" to suit our needs. Lucky for us, we have just the thing: our process-string-adapter.

process-string-adapter will make good on our promise to the handler-fn . Since process-string-adapter calls a business logic function, we can call it a driving adapter, as in, it's driving the process forward. Driving adapters wrap our business logic "interface"(functions) for the system inputs. The business logic here lives in our process-string function which gets called by process-string-adapter (I'm aware process-string does in fact take a string and return a string, it's a contrived example, stay with me here).

process-string has it's own port called logging-fn. logging-fn takes a map and returns nothing, presumably for side-effects. The adapter we supply for our logging-fn port comes in the form of capture-exception-adapter which wraps the "interface" provided by the sentry-clj library. We can call capture-exception-adapter a driven adapter, as in, it's being driven by the process. Driven adapters wrap the "interfaces"(functions) provided by our external services and libraries.

Other Methods

Great, so we can deconstruct protocols into higher-order functions. Instead of passing a list of functions in lieu of protocols, a more realistic alternative to this might be multimethods. It's a rather trivial exercise to replace our functions with multimethods in the example above.

(defmulti log-exception class)

(defmethod log-exception Throwable
  [exception]
  (-> exception
      Throwable->map
      sentry/send-event))

(defmulti process-string class)

(defmethod process-string String
  [input]
  (try
    (apply str (reverse input))
    (catch Exception e
      (log-exception e)
      (throw e))))

(defn handler
  [request]
  (let [input (get-in request [:params :input])]
    {:status 200
     :headers {"Content-Type" "application/json"}
     :body (str "{\"result\": \"" (process-string input) "\"}")}))

(defroutes app-routes
  (GET "/process/:input" request (handlerrequest))
  (route/not-found "Not Found"))

With multimethods we can dispatch on anything, but the better approach might be to dispatch on data. In fact, let's borrow a concept from Onion Architecture: the domain model. Consider a software-as-a-Service solution (SaaS) where a user makes a request to view her business's EBITDA . Her identity, our domain model, is built by the business logic using joins on the users table through an adapter, of course.

{:email "jane.doe@example.com"
 :user-id 1
 :oauth-user false
 :accounting-provider "quickbooks"
 :etc "..."}

Now we can create a multimethod called gross-profit that dispatches on the value :accounting-provider. Dispatch allows the business logic to only care about gross profit for calculating EBITDA provided it has a valid identity. The business logic could conceivably pull gross profit data from any accounting integration added to the system. The multimethod definition gives us a port, and every method creates an adapter for each integration.

(ns best.ebitda.app.gross-profit)

(defmulti gross-profit-for-user
  "A port for getting the gross-profit for a user model.
  Takes a `user-profile` and returns a gross-profit bigdec
  for that user."
  (fn [user-profile]
    (keyword (:provider user-profile))))

(defmethod gross-profit-for-user :default
  [user-profile] 0M)
  
;; in another namespace
(defmethod gross-profit/gross-profit-for-user :quickbooks
  [user-profile]
  (let [{:keys [user-id]} user-profile]
    (some->> user-profile
             :user-id
             (get-quickbooks-account-id)
             (get-quickbooks-gross-profit user-id))))

So far all of these examples live in a single process, but we've created a pathway to a strangler pattern where we can easily strip out our dependencies into separate processes if we so choose.

For more experienced functional programmers, most of this post will seem introductory, but I introduce some more advanced stuff near the end.

Functional Programming over Canadian Programming

Lately, I've been thinking about this Functional Programming journey I've been on. I thought it started when I was twenty, but It may have started when I was twelve. See, I used to watch a lot of G4TechTV here in Canada. I still remember Amber Macarthur introducing a cool new website called Remember the Milk.

Remember the Milk was, to my knowledge, one of the first ToDo list Software-as-a-Service apps, circa 2005. All that tech consumer television certainly inspired me to, not only become a computer nerd, but it inspired me to take up programming at thirteen years old because I believed computing was the future.

I didn't learn functional programming at thirteen years old though. Like most, I started with an imperative programming language. I learned to program with the easiest programming language around: The C programming language. Okay, it wasn't that easy, but I still learned. Making introduction to computer programming in University a breeze.

All throughout University I was still a Remember the Milk user. One day their website spawned a careers page. Much to my curiousity, I decided to checkout the Software Engineer position as I usually did on those kinds of job postings back then. Remember the Milk's job description was different though. There was talk about Scalable Internet Architectures, and this obscure programming language called Scala.

Soon after, I became enamoured with Scala. I thought "this was what Java should have been", but I hit a wall soon after reading Scala for the Impatient. And, I was impatient. I didn't really understand the funcitonal programming features of Scala. Most of my Scala code looked similar to the Java code I was writing in my programming classes. What the hell were closures, reducers, special-access methods, actors, and pattern matching? No fucking clue. Of course my internet research lead me to another functional programming language, a pure functional programming language, Haskell.

Haskell suffered from none of the deficiencies of Java Virtual Machine (JVM) functional programming languages at the time. It had it's own complier that produced binaries as well as having it's own read-eval-print-loop (REPL), and importantly, because I was a broke student, there was a great book online for free to learn Haskell. Finally, I had unlocked functional programming. I went back to Scala, only using the functional programming features for years.

Solving problems with Scala and functional programming eventually qualified me for a full-time Clojure position when I finally dropped out of University.

Transformations over Instructions

The first mindset shift when learning Haskell was to think of a program as a series of transformations on data rather than a series of instructions. A philosophy I would carry into other programming languages throughout my career, like when I was Golang developer. Pure functions and reducers like map and filter made for simplicity and easy testing. Transformations on data are so important, Clojure provides transducers to decouple those transformations from their contexts. Of course, most of us know what a simple series of transformations look like in Clojure:

(->> (range)
     (filter even?)
     (map inc)
     (take 10))

(->> (range)
     (filter even?)
     (map inc)
     (take 10)
     (reduce +))

Thinking of programs as a series of transformations was necessary because functional programming languages defaulted to immutable or persistent data structures. Meaning, if you want to change the data in the structure, you didn't mutate the data structure, rather you got a whole new data structure in memory at the end of the transformation, preserving the old data structure.

At a high level, an operation like map produces a collection the same length as it's input collection, applying an input function to each value in the input collection as the output collection. Simple on the surface. But, how do you do that without state and, therefore, looping?

Recursion over Looping

Pure functional programming languages like Haskell do not contain a looping construct like for while or do...while. Loops, by their nature, often facilitate side-effects and mutate a variable to track whether or not to continue iteration.

Inspired by mathematics (a recurring theme in functional programming), functional programming languages often facilitate iteration through recursion. The mindset shift being, how do we construct a recursive function to emulate a loop?

We can emulate an iteration in a functional programming language by taking advantage of the lexical scope each recursive call has.

(defn recursive-map
  "Recursively applies `f` to each element in `coll`,
  returning the new collection."
  [f coll]
  (let [[head & tail] coll]
    (if (empty? coll)
      nil
      (cons (f head) (recursive-map f tail)))))
      
user> (recursive-map identity (range 10))
(0 1 2 3 4 5 6 7 8 9)
user> (recursive-map inc (range 10))
(1 2 3 4 5 6 7 8 9 10)

Typically, code like this is an anti-pattern in Clojure as it would consume too much stack space for large input. We would have passed our output through each call, building it through the recursive calls like using Clojure's loop and recur. Unfortunately, Clojure does not have tail-call optimization. To get around this, it has a special form recur to signal to the compiler not to consume stack space, but it has to be in the end position unlike our recursive-map call.

I chose the example above because it shows a common functional programming technique: Destructuring our input collection into a head and tail. Head being the first element in our collection, and tail being a collection with all except the head of the collection. Rather than looping over an array and applying our function, each recursive call applies a 1-arity function f to each element before getting cons'd into the list.

Functions over Objects

Our recursive-map above example uses another mindset shift: higher-order functions. Higher-order functions form the basis for everything else in functional programming. In functional programming languages, functions are first class, meaning we can treat them like any other piece of data. We can pass them to other functions as arguments, return functions from functions, and even having lists of functions. Higher-order functions let us decouple a function from it's usage, provided the function signatures align.

user> (take 1 (map println [1 2 3 4 5]))
;; 1
;; 2
;; 3
;; 4
;; 5
;; (nil)
user> (take 1 (map println '(1 2 3 4 5)))
;; 1
;; (nil)

Functional Programming emphasizes pure functions. That is, functions free from side-effects. Side-effects are behaviours functions perform during their execution aside from computing the output. For example, IO operations, network transmission, and writing to a database are all side-effects. Side-effects cause numerous bugs in software development, so by (nearly) eliminating side-effects, functional programming (nearly) elminates this whole class of bugs.

Design over Spaghetti

I like to think of transformations, recursion, and functions as the fundamentals of functional programming. With them we can design and emulate a finite state machine (FSM) without any actual state by using the recurring lexical scope to bind our current FSM state. Whenever the state "transitions", we'll return a function to replace our game-state in game-loop with a new state for our fake game engine below. Even though I wrote this example in Clojure, there's nothing stopping someone from replicating it in Haskell:

(defn pause-menu
  [game]
  (let [{:keys [menu player]} game]
    (do
      (draw-game game)
      (if (= :affirm (:action player))
        (when-let [menu-item (collision player)]
          (cond
            (= menu-item :resume-game) play-game
            (= menu-item :options) (partial options-menu pause-menu)
            (= menu-item :quit-game) quit-game
            :else pause-menu))
        pause-menu))))

(defn options-menu
  [previous game]
  (let [{:keys [menu player]} game]
    (do
      (draw-game game)
      (if (= :affirm (:action player))
        (when-let [menu-item (collision player)]
          (cond
            (= menu-item :controls) controls-menu
            (= menu-item :accessibility) accessibility-menu
            (= menu-item :exit-options) previous
            :else options-menu))
        options-menu))))

(defn play-game
  [game]
  (let [{:keys [components player]} game]
    (do
      (doseq [component components]
        (update! component))
      (draw-game game)
      (if (= :paused (:action player))
        pause-menu
        play-game))))

(defn start-menu
  [game]
  (let [{:keys [menu player]} game]
    (do
      (draw-game game)
      (if (= :affirm (:action player))
        (when-let [menu-item (collision player)]
          (cond ;; our state transition
            (= menu-item :start-game) play-game
            (= menu-item :options) (partial options-menu start-menu)
            (= menu-item :quit-game) quit-game
            :else start-menu))
        start-menu))))

;; Our initial state
(defn start-game
  [game]
  (start-menu game))

(defn game-loop
  [game-state game-engine]
  (when-not (:quit game-engine)
    (recur (game-state game-engine) (game-engine))))

(defn -main
  []
  (game-loop (start-game game-engine) (game-engine)))

Regular readers will recognize this FSM as my "stateless" state pattern from my design patterns series, but I want to emphasis something else: how we can use the fundamentals of Functional Programming to design solutions without making our lives miserable. At least until AI puts us out of work.

Closure over Conclusion

Functional Programming is awesome. When I started learning Haskell, it felt like learning to code all over again. After the initial hurdle, Functional Programming gave me a cognitive model for programming that leads to more simplicity and less bugs by emphasizing transformations, recursion, and functions.

Specifically, from the documentation 'a transducer is a transformation from one reducing function to another'. We can think of a transducer as a context-independent transformation composed of, say, many reducers (like map and filter). If we think of functional programming as a series of transformations on data, we'd naturally want to decouple the transformations from the contexts they're used in. Decoupling our transformations gives us better building blocks for functional programs.

Independence means composition

If transducers are independent of their context, we can also compose transducers into larger transducers instead of only composing reducers. Let's look at an example.

;; Consider a transducer that filters out deleted records and
;; packages them into a format that reagent likes to consume.
(def active-indexed-xf
  (comp (filter #(false? (:deleted %)))
        (map-indexed (fn [idx item] (merge item {:idx idx})))))

(defn list-orders
  "Returns all active orders in a format that Reagent components
  like. Expects pagination query params `page` and `limit` to be
  present in `req`."
  [req]
  (let [results (get-all-orders)
        {:keys [page limit year]
         :or {page 1 limit 25}} (:query-params req)
        page (dec page)
        xfv (cond-> [active-indexed-xf]
              ;; For each optional query param we can just
              ;; keep conj'ing different transducers to the vector to
              ;; further refine the result for our users.
              year (conj (filter #(= (:year %) year)))
              ;; month (conj (filter #(= (:month %) month)))
              (and (not (neg? page)) (pos? limit)) (conj (drop (* page limit))
                                                         (take limit)))
        xf (apply comp xfv)]
    ;; Could also use (apply eduction (conj xfv results))
    ;; or (sequence xf results)
    (into [] xf results)))

Here we can see the classic way of creating a transducer, by using comp to compose many reducers into a single transform, active-indexed-xf. When the list-orders handle is called, our transducer can be comp 'd with other functions that return transducers as well, allowing our API return a list of orders according to the query parameters provided.

We understand how transducers can be composed, but how is it transducers be independent of their context? Well, like most things in Clojure: the higher-order function.

Transducer as a type

As some of you may know, I have a little thing for types. Anything conforming to an interface is a type, and an interface is a bag of functions, or even just one function that has multiple arity, and returns something specific for each. Looking at the code for reducers returning transducers like map, we can see the transducer interface in the 1-arty map function.

;; Copyright Rich Hickey, used under license.
;; map from clojure.core 
(defn map
  "Returns a lazy sequence consisting of the result of applying f to
  the set of first items of each coll, followed by applying f to the
  set of second items in each coll, until any one of the colls is
  exhausted.  Any remaining items in other colls are ignored. Function
  f should accept number-of-colls arguments. Returns a transducer when
  no collection is provided."
  {:added "1.0"
   :static true}
  ([f]
    (fn [rf]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result input]
           (rf result (f input)))
        ([result input & inputs]
           (rf result (apply f input inputs))))))
  ([f coll]
   (lazy-seq
    (when-let [s (seq coll)]
      (if (chunked-seq? s)
        (let [c (chunk-first s)
              size (int (count c))
              b (chunk-buffer size)]
          (dotimes [i size]
              (chunk-append b (f (.nth c i))))
          (chunk-cons (chunk b) (map f (chunk-rest s))))
        (cons (f (first s)) (map f (rest s)))))))
  ([f c1 c2]
   (lazy-seq
    (let [s1 (seq c1) s2 (seq c2)]
      (when (and s1 s2)
        (cons (f (first s1) (first s2))
              (map f (rest s1) (rest s2)))))))
  ([f c1 c2 c3]
   (lazy-seq
    (let [s1 (seq c1) s2 (seq c2) s3 (seq c3)]
      (when (and  s1 s2 s3)
        (cons (f (first s1) (first s2) (first s3))
              (map f (rest s1) (rest s2) (rest s3)))))))
  ([f c1 c2 c3 & colls]
   (let [step (fn step [cs]
                 (lazy-seq
                  (let [ss (map seq cs)]
                    (when (every? identity ss)
                      (cons (map first ss) (step (map rest ss)))))))]
     (map #(apply f %) (step (conj colls c3 c2 c1))))))

Looking the 1-arity call to filter, we can see something similar.

;; Copyright Rich Hickey, used under license.
;; filter from clojure.core
(defn filter
  "Returns a lazy sequence of the items in coll for which
  (pred item) returns logical true. pred must be free of side-effects.
  Returns a transducer when no collection is provided."
  {:added "1.0"
   :static true}
  ([pred]
    (fn [rf]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result input]
           (if (pred input)
             (rf result input)
             result)))))
  ([pred coll]
   (lazy-seq
    (when-let [s (seq coll)]
      (if (chunked-seq? s)
        (let [c (chunk-first s)
              size (count c)
              b (chunk-buffer size)]
          (dotimes [i size]
              (let [v (.nth c i)]
                (when (pred v)
                  (chunk-append b v))))
          (chunk-cons (chunk b) (filter pred (chunk-rest s))))
        (let [f (first s) r (rest s)]
          (if (pred f)
            (cons f (filter pred r))
            (filter pred r))))))))

Now for the last piece of the puzzle, look at how comp composes functions.

;; Copyright Rich Hickey, used under license.
;; comp from clojure.core
(defn comp
  "Takes a set of functions and returns a fn that is the composition
  of those fns.  The returned fn takes a variable number of args,
  applies the rightmost of fns to the args, the next
  fn (right-to-left) to the result, etc."
  {:added "1.0"
   :static true}
  ([] identity)
  ([f] f)
  ([f g] 
     (fn 
       ([] (f (g)))
       ([x] (f (g x)))
       ([x y] (f (g x y)))
       ([x y z] (f (g x y z)))
       ([x y z & args] (f (apply g x y z args)))))
  ([f g & fs]
     (reduce1 comp (list* f g fs))))

I'm sure some of you have figured out that calling comp on transducers will give us nested fn calls.

;; Doing this
(comp (filter even?)
      (map inc))
      
;; Gives us something like this:

;; rf is reducing function here
;; if we passed this xf to transduce with `+`, + is our reducing fn
;; rf from filter, is now bound to the function returned by 1-arty map
(fn [rf]
  (fn
    ([] (;; rf from map. I've sustituted the rf from filter for what
         ;; it is bound to, the function from map.
         (fn [rf]
           (fn
             ([] (rf))
             ([result] (rf result))
             ([result input]
              (rf result (f input)))
             ([result input & inputs]
              (rf result (apply f input inputs)))))))
    ([result] ((fn [rf]
                 (fn
                   ([] (rf))
                   ([result] (rf result))
                   ([result input]
                    (rf result (f input)))
                   ([result input & inputs]
                    (rf result (apply f input inputs))))) result))
    ([result input]
     (if (pred input)
       ((fn [rf]
          (fn
            ([] (rf))
            ([result] (rf result))
            ([result input]
             (rf result (f input)))
            ([result input & inputs]
             (rf result (apply f input inputs))))) result input)
       result))))

Passing a transducer to another transducer "stacks" the operation on top of one another, revealing the interface for transducers. A transducer is a function that takes a reducing function ( rf ) and returns a closure over it with three arities.

• Init, or arity zero, calls the init arity on the rest of the stack.
• Step, or arity two, calls the reducing function on two values, like a single step of reduce.
• Completion, or arity one, Cleans up stateful transducers and returns a value. It must call the reducing function completion once.

If even? is true, then the value will be incremented in our example before being passed to the reducing function + in transduce.

(def xf (comp (filter even?) (map inc)))
(transduce xf + (range 100))

In a sense transducers 'parallelize' multiple transformations from stacking them on top of one another.

Gotta Go Fast

For most, developers often think of the parallelization as a forgone conclusion. I don't blame them. Every Computer Science faculty in the world (I hope) teaches about computational complexity analysis of algorithms like filter and map. They're both average time complexity of O(n), or linear time. Meaning, the algorithms time to run will be a linear function of it's input. So, according to Big-O Notation, even if our input is 5n, our time complexity will still be on the order of O(n).

Unfortunately for us working outside of the classroom, we have cloud bills to worry about. If we're working with the massive input of a collection with, say, five million records, we might not have the computational luxury of naively threading ( ->>) through a bunch of reducers. Even though crunching one million records and five million records have the same time complexity for clojure.core's reducers, in reality the latter has five times more instructions to execute sequentially with threading macros. It would be far more efficient if we just parallelize them using transducers!

Let's do a benchmark with Criterium to confirm.

user> (criterium.core/with-progress-reporting
        (criterium.core/bench
         (into []
               (comp
                (map inc)
                (filter odd?)
                (map dec)
                (filter even?)
                (map (fn [n] (+ 3 n)))
                (filter odd?)
                (map inc)
                (filter odd?)
                (map dec)
                (filter even?)
                (map (fn [n] (+ 3 n)))
                (filter odd?))
               (range 5000000))))
Warming up for JIT optimisations 10000000000 ...
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
Evaluation count : 600 in 60 samples of 10 calls.
             Execution time mean : 105.381854 ms
    Execution time std-deviation : 900.473165 µs
   Execution time lower quantile : 103.346469 ms ( 2.5%)
   Execution time upper quantile : 106.385211 ms (97.5%)
                   Overhead used : 1.598527 ns

Found 4 outliers in 60 samples (6.6667 %)
	low-severe	 3 (5.0000 %)
	low-mild	 1 (1.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
nil
user> (criterium.core/with-progress-reporting
        (criterium.core/bench
         (->> (range 5000000)
              (map inc)
              (filter odd?)
              (map dec)
              (filter even?)
              (map (fn [n] (+ 3 n)))
              (filter odd?)
              (map inc)
              (filter odd?)
              (map dec)
              (filter even?)
              (map (fn [n] (+ 3 n)))
              (filter odd?)
              (into []))))
Warming up for JIT optimisations 10000000000 ...
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
Evaluation count : 240 in 60 samples of 4 calls.
             Execution time mean : 256.346727 ms
    Execution time std-deviation : 1.908808 ms
   Execution time lower quantile : 254.553894 ms ( 2.5%)
   Execution time upper quantile : 258.981447 ms (97.5%)
                   Overhead used : 1.598527 ns

Found 4 outliers in 60 samples (6.6667 %)
	low-severe	 1 (1.6667 %)
	low-mild	 2 (3.3333 %)
	high-mild	 1 (1.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
nil

Quite a reduction in execution time when using transducers on large input! You won't see that if the input was (range 10) and (range 50) though, there's just not enough to execute sequentially to see a worthy jump in performance.

Conclusion

In my opinion, transducers are an underrated feature in Clojure. Not only do we get functional programming building blocks for little cost, we get a signifigant boost in performance when working with large datasets.

I'm sure for some of you Model-View-Controller (MVC) resembles more of an anti-pattern than something to reach for when writing Clojure, but I think it's a great example of what architecture should accomplish, especially what can be accomplished in Clojure.

MVC decouples state, state changes, and state usage. One way to compose this architecture uses three classic software design patterns: Strategy as our controller to update the model, observer to notify the view and controller, and composite as our view hierarchy.

If we want to do this in Clojure, we can use the Clojure design pattern analogs. Let's use a higher-order functions for our strategy; a watch for our observer (watching an atom or ref); and seesaw components for our composite, we can build ourselves a simple desktop application: an adder.

The Controller

Since for the purposes of this example we'll pretend we're purists, our strategy will be a couple higher-order functions (no protocols).  So let's create a couple delegate functions to update our model.

;; Higher-order functions as our "strategy delegates".
(defn change-display
  "Takes an atom `model` and a string `input`, and updates
  `model` by appending `input` to the currently displayed
  text in our calculator."
  [model input]
  (swap! model update :display #(apply str (concat % input))))

(defn solve
  [model]
  (swap! model update :display #(apply + (mapv (fn [s]
                                                 (Integer/parseInt s))
                                               (string/split % #"\+")))))

I understand in the land of YAGNI, this seems more like set dressing with code complexity than something useful, but it has a usefulness in a bigger code base as we're about to see how it decouples state changes from the using the state.

The View

I never wrote about the composite pattern in Clojure (not yet anyway, subscribe for design patterns part 3). To summarize, the composite pattern creates an interface for building hierachies with the intent that children in the hierarchy will follow the interface and propagate changes down the tree. We can do this pretty easy in Clojure using higher-order functions (again) or protocols, but for now we'll rely on Seesaw, a wrapper around Java's Swing components.  

;; A very simple view hierarchy using a few seesaw components.
(defn create-view-children
  [display state change-fn solve-fn]
  (let [display-label (s/label :text display)
        plus-button (s/button :text "+"
                              :listen [:action (fn [e]
                                                 (change-fn state "+"))])
        equal-button (s/button :text "="
                               :listen [:action (fn [e]
                                                  (solve-fn state))])
        keypad-buttons (mapv (fn [n]
                               (s/button :text (str n)
                                         :listen [:action (fn [e]
                                                            (change-display state (str n)))]))
                             (range 9))
        keypad (s/grid-panel :rows 3
                             :columns 3
                             :items keypad-buttons)
        panel (s/grid-panel :rows 1
                            :columns 2
                            :items [plus-button equal-button])]
    (s/border-panel
     :items [[panel :south] [display-label :north] [keypad :center]])))

(defn create-view
  [children]
  (s/frame :title "Positive Integer Adder"
           :visible? true
           :content children
           :on-close :exit
           :width 400
           :height 300))

For our view, we use map to programmatically create our number buttons in the  hierarchy as well as directly creating plus and equals buttons. Storing them in a couple panels to describe the layout of the adder's UI. We will pass these components to our view factory function create-view which will set up the Swing frame for us. Now all we need to do is update our view everytime the model's state changes to update our display-label.

The Model

Lastly, we see our model and it's observer are pretty simple. Just an atom and a watch. I added the state-logger function to help debug the output, but we could add any number of responses to the state changes here without affecting the view (unless, of course, we removed or changed update-view).

;; We can add many "observers" to change other view state/composition
;; or even change the strategy used by the buttons to add the numbers now
;; that we've decoupled all three.
(defn update-view
  [view state]
  (fn [_ _ _ new-state]
    (let [display (:display new-state)]
      (s/config! view :content (create-view-children display
                                                     state
                                                     change-display
                                                     solve)))))

(defn state-logger
  [_ _ _ new-state]
  (println "State changed to: " new-state))

(defn -main
  [& args]
  (let [;; Our model, diligently keeping track of our state
        state (atom {:display ""})

        ;; The root of our view hierachy
        view (create-view (create-view-children 0 state change-display solve))]

    ;; Observe
    (add-watch state :update-view (update-view view state))
    (add-watch state :logger state-logger)

    ;; Start the app
    (s/native!)
    (s/invoke-now (fn [] (s/show! (s/pack! view))))))

As the comments suggest, we can even change the controller or view we want to use. Then, we put it all together in -main, creating our model (atom), view (seesaw), and start watching the show unfold.

Putting it all together

I'm guessing you've already put all the code examples together, but let me help you out by doing just that.

(ns janetacarr.mvc-adder
  (:require [seesaw.core :as s]
            [clojure.string :as string])
  (:gen-class))

;; Higher-order functions as our "strategy delegates".
(defn change-display
  "Takes an atom `model` and a string `input`, and updates
  `model` by appending `input` to the currently displayed
  text in our calculator."
  [model input]
  (swap! model update :display #(apply str (concat % input))))

(defn solve
  [model]
  (swap! model update :display #(apply + (mapv (fn [s]
                                                 (Integer/parseInt s))
                                               (string/split % #"\+")))))

;; A very simple view hierarchy using a few seesaw components.
(defn create-view-children
  [display state change-fn solve-fn]
  (let [display-label (s/label :text display)
        plus-button (s/button :text "+"
                              :listen [:action (fn [e]
                                                 (change-fn state "+"))])
        equal-button (s/button :text "="
                               :listen [:action (fn [e]
                                                  (solve-fn state))])
        keypad-buttons (mapv (fn [n]
                               (s/button :text (str n)
                                         :listen [:action (fn [e]
                                                            (change-display state (str n)))]))
                             (range 9))
        keypad (s/grid-panel :rows 3
                             :columns 3
                             :items keypad-buttons)
        panel (s/grid-panel :rows 1
                            :columns 2
                            :items [plus-button equal-button])]
    (s/border-panel
     :items [[panel :south] [display-label :north] [keypad :center]])))

(defn create-view
  [children]
  (s/frame :title "Positive Integer Adder"
           :visible? true
           :content children
           :on-close :exit
           :width 400
           :height 300))

;; We can add many "observers" to change other view state/composition
;; or even change the strategy used by the buttons to add the numbers now
;; that we've decoupled all three.
(defn update-view
  [view state]
  (fn [_ _ _ new-state]
    (let [display (:display new-state)]
      (s/config! view :content (create-view-children display
                                                     state
                                                     change-display
                                                     solve)))))

(defn state-logger
  [_ _ _ new-state]
  (println "State changed to: " new-state))

(defn -main
  [& args]
  (let [;; Our model, diligently keeping track of our state
        state (atom {:display ""})

        ;; The root of our view hierachy
        view (create-view (create-view-children 0 state change-display solve))]

    ;; Observe
    (add-watch state :update-view (update-view view state))
    (add-watch state :logger state-logger)

    ;; Start the app
    (s/native!)
    (s/invoke-now (fn [] (s/show! (s/pack! view))))))

If we run this code with our Clojure CLI, we'll see this window pop up:

Hitting equals at this point will call our solve strategy, updating the model to contain the string "22" and, of course, the display in the application will also show 22 instead of 11+11.

Conclusion

I understand coding an MVC app under one hundred lines of code seems a bit trite, but we can bridge the mental gap here to something like Rails or other web frameworks by replacing our seesaw components with serving HTML fairly easily. Don't be so quick to dismiss the potential of a little state and some higher-order functions.

One day my squad mate at a Clojure startup committed a macro to the codebase, and I was quick to reprimand them for it as the same functionality could have been implemented with Clojure core functions. Peeling back the decision rationale, I discovered my colleague had never learned the first and second rule of Clojure macros: Don't talk about Clojure macros. Just kidding, the real rule is you should only reach for a macro if Clojure doesn't already support something you need (or if you need to stop evaluation, but that's another story).

So what is a macro anyways? A macro is code that operates on other code, and typically they return code as their result. But what do all the tildes, at-symbols, pound (#), and back ticks mean?

Data is beautiful, and so is code?

It's no coincidence Clojure forms (function calls) share a similar syntax to Clojure's list type. Clojure forms are lists! Every Clojure developer has made this mistake (and, some tired days, many times):

user> (1 2 3)
Execution error (ClassCastException) at user/eval79101 (dev.clj:82).
class java.lang.Long cannot be cast to class clojure.lang.IFn (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.IFn is in unnamed module of loader 'app')                                                                          
user> '(1 2 3)                                                                                  
(1 2 3)

Since Clojure forms are just lists, we can treat Clojure code like data because it is data. What happens if we use that little quote ' on a function? Check it out:

user> '(defn hello-world [] (println "Hello, data! Or code??"))
(defn hello-world [] (println "Hello, data! Or code??"))
user> (eval '(defn hello-world [] (println "Hello, data! Or code??")))
#'user/hello-world
user> (hello-world)
Hello, data! Or code??
nil
user> 

We get a list! Specifically, we get a list starting with two symbols, defn and hello-world, followed by an empty vector, trailing with another list containing the symbol println and a string. With such a list, we can pass it to eval to get our function. eval evaluates its input into something useful, whether it's a map or a function. This reveals the heart of our macro, we simply return code to be evaluated and used.

But there's a problem with the previous code though. We can't really manipulate Clojure code like this because all the contents of the list get evaluated on their own. If we want to operate on code, we have to do it before the Clojure compiler evaluates our code.

All the cool compilers Read

A lisp compiler has a stage before evaluation called reading. A program called the reader reads the code from source files and then evaluates it. Lucky for us we can manipluate code before evaluation with a macro! In this sense, macros are like special functions for the reader, and Clojure provides us with a special form for creating macros called defmacro.

Using defmacro, the def- macro below returns a list of symbols and lists, so evaluation can turn it into a handy form that binds a symbol to decls with the private metadata flag set to true so functions outside this namespace can't access the symbol's value.

(defmacro def-
  "same as defn-, yielding non-public def"
  [name & decls]
  (list* `def
         (with-meta name
           (assoc (meta name) :private true))
         decls))

Notice the backtick in front of def in our def- macro? Backtick in Clojure is called Syntax Quote, and works like the quote but will fully resolve symbols for us. In other words, it'll find the symbol's namespace for us. For example, `def yields clojure.core/def . Without syntax quote our def is just a plain symbol devoid of any value in the symbol table to the Clojure reader.

user> `(defn hello-world [] (println "Hello, data! Or code??"))
(clojure.core/defn
  user/hello-world
  []
  (clojure.core/println "Hello, data! Or code??"))
user> 

So to tell the reader, "hey look here for this thing", we need to give it directions to something already in the symbol table by using the fully qualified symbol (a symbol with a namespace prefix).  We have to fully qualify our symbols in macros because namespace aliases don't exist to the reader. Normally an alias might look like using (:require [foo.bar :as foob]) at the top of your Clojure source file. Because the Clojure compiler automatically imports all of clojure.core to every source file, we still have to account for it as an alias in the form (:require [clojure.core :refer :all]) .

Your favorite Unquote?

You'll almost always see Syntax Quote close to ~, or unquote, in Clojure macros. We can think of it as an operator to evaluate just this one thing. Similarly, we'll see ~@ , or unquote splice, and we can think of unquote splice as an operator to pull data out of a list. This example from the Clojure documentation demonstrates the use of syntax quote, unquote, and unquote splicing pretty well:

user=> (def x 5)
user=> (def lst '(a b c))
user=> `(fred x ~x lst ~@lst 7 8 :nine)
(user/fred user/x 5 user/lst a b c 7 8 :nine)

Unquote and unquote splice are instrumental to writing macros where we can add our own features to Clojure as these give us the ability to have values, and unfurl lists. So putting those together, we can get the gist of something like when-let :

(defmacro when-let
  "bindings => binding-form test
  When test is true, evaluates body with binding-form bound to the value of test"
  {:added "1.0"}
  [bindings & body]
  (assert-args
     (vector? bindings) "a vector for its binding"
     (= 2 (count bindings)) "exactly 2 forms in binding vector")
   (let [form (bindings 0) tst (bindings 1)]
    `(let [temp# ~tst]
       (when temp#
         (let [~form temp#]
           ~@body)))))

Looking at when-let , we can see this macro takes a vector of bindings and body of forms in a multivariant argument list. The first let takes the first and second values out of the bindings, just like we'd expect from a let but then passes the second binding, tst to syntax quoted let, setting it to a temporary binding called temp# . temp# gets used as the test for our when macro where temp# gets bound again to the first value (a simple symbol) in our original bindings. Finally, body gets unquote spliced to take out all the forms in the list body is and passed to an implicit do in the closest let . The implicit do is very important as we'd just be returning a list of forms again. If there wasn't an implicit one, you'd have to use an explicit one.

We can expand our use of when-let using macroexpand-1 and macroexpand to see the first step and all steps in macro expansion, respectively.

user> (macroexpand-1 '(when-let [cool "hi"] (println cool)))
(clojure.core/let
    [temp__5804__auto__ "hi"]
  (clojure.core/when
      temp__5804__auto__
    (clojure.core/let [cool temp__5804__auto__] (println cool))))
user> (macroexpand '(when-let [cool "hi"] (println cool)))
(let*
    [temp__5804__auto__ "hi"]
  (clojure.core/when
      temp__5804__auto__
    (clojure.core/let [cool temp__5804__auto__] (println cool))))
user> 

Hey wait, what the hell is temp__5804__auto__? Remember early when I talked about how the reader has no concept of aliasing? If we want to use simple (unqualified) symbols, we also have the potential for symbols having the same names (symbol collision), where ever the macro is used. To avoid symbol collision the hash used at the end of temp# in a macro tells the reader to generate a symbol for us. The hash will only generate one unique symbol though. If we wanted to have a recursive macro with a unique symbol for each recursive call, we call the gensym function to generate a unique symbol each call.

Too long; didn't read

Macros are a cool, powerful feature of Clojure. While they are great fun to write, they are often discouraged in codebases because of the complexity of using code to manipulate other code. Often, this makes macros a nightmare to maintain and debug. The goal of this post is to give you the confidence and tools to read and understand macros as Clojure macros have been the subject of whole books.

A quick summary:

• Macros are code that operates on other code, and are used in the read stage
• Clojure forms are really just lists
• Get a list with simple symbols by prefixing our function call with quote or '
• Get a list with fully qualified symbols by prefixing our function or list with a syntax quote  `(defn hello-world ...)
• Evaluate something using unquote or ~ in a macro
• Pull out the contents of a list using unquote splicing or ~@
• A symbol postfixed with a # will yield a unique symbol by the syntax reader
• If you need a unique symbol for each recursive macro expansion, use the gensym function.

And that's it, I hope you learned enough to feel confident working with macros.

Ever since I wrote Clojure needs a Rails, folks reach out to ask for my opinion on their web framework for Clojure. What's clear from these discussions is a lot of software developers have no idea what a framework is. Frameworks follow a principle called "Inversion of Control", where developers delegate crucial decisions for their application to the framework maintainers, allowing the developers to deliver the application faster. It is not a project template, a new paradigm, or module system.

More Decisions, More Problems

Often, developers suffer from decision paralysis and excessive engineering prowess. I understand why developers are like this. I was like this too (see my design patterns posts for years of pent up engineering hubris). Unfortunately, we are in the business of solving problems and making money with those solutions, so the sooner the better. We get there sooner by not bothering with things like architecture, databases, configuration, convention, and tooling, in other words, we can just get on with the coding.

Architecture

Arguably, the most important decision delegated to your framework is the software architecture, and many consider it the defining feature of a Software Framework, otherwise, you might just be using a software developer kit (SDK), bundle of libraries, or some toolkit. All of those are fine to use, but I doubt they'll be your next Rails.  

Database

How frameworks handle databases tell me if you mean business. I don't care about how it gets done, honestly. I want a defentity macro (or something) and through the power of macro magic we get CRUD handlers (controllers?) for this entity, a dashboard to manipulate them, and hooks to control access. The core idea here being abstraction, why do we have to write the same queries over and over again? Most databases can be abstracted away in a data-mapper or ORM, I know those are dirty words to the Clojure community, but something truly great would give us the same powerful abstraction to create database things while allowing us to choose our database with an opinated plugin.

Tooling

Our toolchain and libraries fit the mould, our opinated MySQL or Datomic plugin just works, because it's made by the framework maintainers, and allows the same macro magic to create our entities. Notice how these plugins don't have the caveat of  "you can just replace the whole thing with modules/plugins!", if I wanted to do something like that, I would, yet again, start a project from scratch withdeps.edn and the Clojure CLI. A bundle of libraries and glue code can masquerade as a framework without direction or purpose, so beware, you might end up configuring your whole stack anyways.

Configuration

EDN files, database migrations, environment variables, feature flags, software versioning, docker compose, what the hell are we doing with our time. Ask any Site Reliability Engineer(ing specialist for legal purposes), and she or he will tell you that configuration mishaps are responsible for a vast majority of incidents and outages, and we're still writing them all by hand over and over again. One snake_case name, where it should have been kebab-case, and now VP of Engineering is breathing down our Zoom call necks.

Convention

Thanks to Clojure's dependence on camelCase virtual machines and their libraries, sometimes our conventions breaks. Hell, sometimes developers just don't care, yet developers make decisions every second of writing code and rely on those conventions to ship a functioning product. The cognitive overhead of using the REPL to dig into a software rotted library to see they keywordized the camelCase keys from JSONland not bothering to transform them to our beloved kebab-case (or worse, mixing them) causes bugs and slowdown.

Opinions are good, actually

If the past several paragraphs were any indication, opinions are good. They might not be scalable to every project, but I think there will be opinions to fill the void at some point. I'm writing this article because I see an awful lot of scoffing at the idea of something opinionated in the Clojure community. If you don't want something opinated, that's fine. Personally, I do not have the ego to say I'm smarter than every other developer, so let me rewrite everything that has ever been written once again. So let me ask you dear reader, does this really command the pay of an expensive, paren wielding, high-value producing, knowledge worker?  

Here's the thing,

I see this a lot from clients, employers, hackers, lovers, and haters. Clojure's dynamic typing is a feature, not a bug. If you've ever written a Python script, you know what I'm talking about, but Clojure takes it much further, giving us rare occasion to check types. We can ship code without static typing, we can provide guard rails for our programs, and we can do it without an alpha version.

Programs get built

Yes, we can ship production code with Clojure. A strange bit of grumbling I hear from developers directs their angst toward not having "compile time type checking errors" when using Clojure, leading them to simply chalk up Clojure to being unusable (?). Clojure ships with a lot similar data structures as Java, for example, you can still shoot yourself in the foot with a java.util.HashMap as the get(key) method still returns null if the value doesn't exist. Now, if you're one of those static programming language developers opting to use something more familiar like  a class/struct/union/process, Congratulations. I think you'll find a degree of rigidity not present in Clojure. Rigidity = friction and friction = lower velocity. I like to ship code, and I'm usually pretty confident my Clojure code will work because I have the REPL and an endless amount of parameter checking libraries.

We all have boundaries

I'm not talking about those boundaries! (although, do respect them, please). I'm talking about system boundaries. It's no secret I dislike the over zealous use of Clojure.spec and friends, and I hate the friction it adds development. So in the past I've recommended using it on your system boundaries, if you must. Lately, for my web apps, I've been using Malli as the type checking and coercion for RESTful APIs routed with Reitit. It's great. After that, the gloves come off, because spending the overhead of 'specing' on top of REPLing and testing Clojure code, I find Clojure.spec doesn't add much value for the time cost. Lucky for us, we get some goodies from Clojure to help with type woes. Clojure's collections, vectors, maps, and sets (and nil?), all support the same handful of functions. I've talked about how a type or interface is really a bag of functions. Replacing an indexed value in a Clojure vector is a surprisingly annoying task, but I think most developers don't realize you can actually call assoc on a vector using the index as a key (or at least I don't see it in my Clients' codebases much).

user> (assoc (vec (range 10)) 3 5)
[0 1 2 5 4 5 6 7 8 9]
user> (defn replace-i [v i value] (into (conj (vec (take i v)) value) (drop (inc i) v)))
#'user/replace-i
user> (replace-i (vec (range 10)) 3 5)
[0 1 2 5 4 5 6 7 8 9]

I'm sure some people will see this as just function overloading, but I want you to know that they're god damned liars. I like to think of this as the "do nothing" option, and I have never experienced serious typing problems with Clojure. I have experienced issues with Clojure.spec itself though because Clojure.spec suffers from terminal alpha versioning.

The versioning

Since its inception, Clojure.spec remains in alpha. I'll admit the stability of Clojure.spec is relatively good since it's debut. However, I do think this a sore spot, implying Clojure.spec should not be used in production, even though everyone does.

We can ship production code with Clojure, and we don't need a compiler to tell us to write our code. (But we might need one for Clojurescript ;)

Thanks for reading. Usually this is where I invite people to discuss what I've written here, but honestly I felt like this post was a bit of a dump of half-baked ideas and passion for my Clojure flow state, so maybe just subscribe for the next one,  follow me on twitter @janetacarr , or don't ¯\_(ツ)_/¯ .

Patterns

• Preamble
• Builder
• Chain of Responsibility
• Proxy
• Adapter
• Template Method
• Flyweight
• Strategy

Our story continues

Edmund, a bright, awkward young man with a masters degree from a prestigious University, loves functional programming, but his views are too pure for Suzi's taste. He always drones on about how we don't need to think about any of those design patterns things because "that's only for object-oriented programming".

However, Edmund's latest project with an intern, a load testing application, was taking far too long, like most projects at StartupAI. Despite the mythical (wo)man-month, Suzi's manager asks her to join the project, and she agrees.

"We see the heart of this project as state machine" Edmund says, during a code walk through.

To Suzi surprise, she sees a dense case form. "Uh oh" thinking to herself.

Next Edmund shows her the dozens of functions created to manufacture types to kick start the state machine. Of course, whenever they have to change these functions, they have to change the state machine and vise-versa. To top it all off, they're using Clojure futures to generate the load tests, attempting to coordinate all the futures and their requests. Lucky for Suzi, she knows why they're not progressing on the project, and why it doesn't work at all.

"I have an idea, why don't you replace this case form with a state pattern, and these functions with a builder?"  Suzi says gently presenting the idea.

"Oh, I just prefer not to do those kind of things in functional programming." He replies.

"Right, but it would decouple the state machine from the this namespace, and you can decouple your functions here from this code down here." She says while pointing with her pen at the functions on the screen from earlier, and gesturing downward at the state machine. Continuing, "And, you can ditch this whole futures thing. Just use core.async".

"What's core.async?" Edmund asks no one in particular while typing 'clojure core async' into the browser search bar.

"Communicating Sequential Processes in Clojure. It handles thread pooling and everything, all we have to do is dispatch work to the 'go loops'." Suzi answers.

Oh.

Builder

Separate the construction of a complex object from its representation so that the same construction process can create different representations.

A lot of patterns become thinking about, and structuring of, the flow of higher-order functions and their composition. While the functions may seem vague, we need to keep in mind our function's intent. With the builder pattern we want to delegate object (or function) construction (composition) .

When to use it

When you want to decouple the construction of a complex object from the context it's used in.

Clojure analogue

• partial or fn and comp - Most design patterns are a response to missing features from functional programming. The command pattern address the lack of higher order functions in Object Oriented programming languages. So in this respect, the builder pattern addresses the lack of currying. Unfortunately, Clojure also doesn't support currying, but we can get a close approximation with partial function composition using partial or fn and comp.
• cond-> and assoc and related map functions - In practice most "builders" are really just methods to set some kind of member values rather than delegating object construction. We see this a lot in something like Go where a struct has receivers that return a new struct, allowing for chaining of call like MyCoolStruct.WithFun("stuff").WithHydration("Water").WithCode("Clojure"). If you're idea of a builder looks like this, you can probably get pretty far with threading macros and map functions like assoc, update, merge since they all take a hash map as an argument and return a hash map, or just make up your own functions!

Keep in mind

• Partial function composition can get hairy since you can't name the function produced by either partial or comp.
• When using partial and comp, take care to remember that transducers exist. Depending on your use case composing a transform from transducers (map, filter, etc) will be much more performant than creating partial functions over them.

Sample code

;; builder
;; now we can delegate our map construction to this
;; function.
(defn build-s3-client
  [mock-creds ssl default-bucket sts]
  (let [if-fn (fn [cond f] (if cond f identity))]
    (comp
     (if-fn (not (empty? creds))
       #(let [secret (s/gen :specs/client-secret)
              access-key (s/gen :specs/access-key)]
          (merge % {:client-secret (g/generate secret)
                    :access-key (g/generate access-key)})))

     (if-fn ssl #(assoc % :ssl true))

     (if-fn (and (string? default-bucket)
                 (not (empty? default-bucket)))
       #(assoc % :default-bucket default-bucket))

     (if-fn (and (string? sts)
                 (not (empty? sts))
                 (some? (re-matches #"." sts)))
       #(assoc % :sts-grant sts)))))
   
(def s3-client 
  (build-s3-client {} false "my-bucket" "real-token"))

;; Chaining technique
(defn ->s3-client
  []
  (let [{:keys [aws-secret-key
                aws-access-key]} env]
    {:client-secret client-secret
     :access-key access-key}))

(defn with-ssl
  [m]
  (assoc m :ssl true))

(defn with-default-bucket
  [m bucket]
  (assoc m :default-bucket bucket))

(defn with-sts
  [m token]
  (assoc m :sts-grant token))

(defn with-mock-credentials
  [m]
  (let [secret (s/gen :specs/client-secret)
        access-key (s/gen :specs/access-key)]
    (merge m {:client-secret (g/generate secret)
              :access-key (g/generate access-key)})))

(defn build-s3-client
  [bucket]
  (-> (->s3-client)
      (with-ssl)
      (with-default-bucket bucket)))

(defn mock-s3-client
  [bucket]
  (-> (->s3-client)
      (with-mock-credentials)
      (with-bucket bucket)))

Chain of Responsibility

Avoid coupling the sender of a request to its receiver by giving more than one object a chance to handle the request. Chain the receiving objects and pass the request along the chain until an object handles it.

The structure of this pattern seems similar to the builder, but our intent for our functions is different. And, If I'm being honest, the only Chain of Responsibility I've ever seen, even outside of Clojure, is the middleware chain in ring applications.

When to use it

When you want to decouple a request from its handler.

Clojure analogue

• -> or ->> and fn - Using the thread macro through lambdas that have the same function signature, creating closures over closures until its final form.

Keep in mind

• Chain of Responsibility in Clojure has the same caveats as the command pattern from the last post. Be sure to name your fn forms because the stack traces from the Chain of Responsibility are challenging to debug as the chain can sometimes be quite long.
• Like all threading macros, If the chain order matters, you will have to structure your handlers in reverse order. For example, in the case of ring middleware you want the function returned by wrap-authentication to evaluate before the function returned by wrap-authorization.

Sample code

Ring middleware gives us the best and most prevalent example of a Chain of Responsibility in Clojure. In this example wrap-authentication and wrap-authorization are adapted from buddy.auth.

(defn wrap-authentication
  "Ring middleware that enables authentication for your ring
  handler. When multiple `backends` are given each of them gets a
  chance to authenticate the request."
  [handler & backends]
  (fn authentication-handler
    ([request]
     (handler (apply authentication-request request backends)))
    ([request respond raise]
     (handler (apply authentication-request request backends) 
              respond 
              raise))))
    
;; lots of code in between. Just pretend it's here for
;; the sake of this example.

(defn wrap-authorization
  "Ring middleware that enables authorization
  workflow for your ring handler.
  The `backend` parameter should be a plain function
  that accepts two parameters: request and errordata
  hashmap, or an instance that satisfies IAuthorization
  protocol."
  [handler backend]
  (fn authorization-handler
    ([request]
     (try (handler request)
          (catch Exception e
            (authorization-error request e backend))))
    ([request respond raise]
     (try (handler request respond raise)
          (catch Exception e
            (respond (authorization-error request e backend)))))))
     
;; In another namespace:
;; Middleware usage
(def app
  (-> compojure-routes
      (wrap-authorization jwe)
      (wrap-authentication jwe)
      (wrap-cors :access-control-allow-origin [#".*"]
                 :access-control-allow-methods [:get :post])
      (wrap-json-response {:pretty false})
      (wrap-json-body {:keywords? true})
      (wrap-accept {:mime ["application/json" :as :json
                           "text/html" :as :html]})
      (wrap-cookies)
      (wrap-params)))

Proxy

Provide a surrogate or placeholder for another object to control access to it.

Unlike the only Chain of Responsibility I've seen, I have seen a proxy once or twice. Usually developers don't use the proxy form, rather opting to wrap a function in another function to restrict access, lazy typing, or virtualization.

When to use it

The Gang of Four quote is pretty legible on this one. Use a proxy as a 'middleman' for (controlling) access to another object. You can also use it for creating a lazily loaded object if a LazySeq doesn't do it for you.

Clojure analogue

• proxy - Clojure ships with a whole set of proxy-related functions. They are seldom used as developers typically prefer to use reify(and related forms). Unlike reify, proxycan be used to instantiate Java classes, not just interfaces and protocols.
• You could use higher-order fn to have a proxy too, but at this point the idea is getting repetitive, so I'm just going to keep it to the quirky, interesting stuff.

Keep in mind

• The proxy functions proxy, proxy-super, init-proxy, update-proxy, construct-proxy, get-proxy-class, and proxy-mapping are old (Clojure 1.0) and reminiscent of Object-oriented programming. If you didn't know what a proxy was, you might end up doing double the work of plain old interop when using proxy.
• Some of the proxy functions like proxy-super expand to use reflection on their this argument unless type hinted.
• Like all extensibility on the JVM, proxy can't inherit from a final class which also means no proxies to Clojure records.
• You can't proxy a proxy, meaning you can't use proxy on something that already implements Clojure's IProxy interface. If you could, that'd be OOP. This isn't Common Lisp.

Sample code

(defn access-controlled-input-stream
  "Returns an access controlled InputStream"
  [user]
  (proxy [java.io.InputStream] []
    (read
      ([]
       (if (allowed? user)
         (let [^java.io.InputStream this this]
           (proxy-super read))
         (throw (ex-info "Unauthorized"
                         {:user user}))))
      ([^bytes b]
       (if (allowed? user)
         (let [^java.io.InputStream this this]
           (proxy-super read b))
         (throw (ex-info "Unauthorized"
                         {:user user}))))
      ([^bytes b off len]
       (if (allowed? user)
         (let [^java.io.InputStream this this]
           (proxy-super read b off len))
         (throw (ex-info "Unauthorized"
                         {:user user})))))))

Adapter

Convert the interface of a class into a another interface clients expect. Adapter lets classes work together that couldn't otherwise because of incompatible interfaces.

You don't need to always use a function to wrap something, you can also use an adapter. Adapters are the ultimate glue code and de facto polymorphism (protocols) in Clojure. If we consider a type an interface (or bag of functions™), any library or API could be considered a interface, so we can wrap it in an adapter for core business logic for when we want to swap it out¹. Or not, if you're an "aren't going to need it" person.

When to use it

When you want to use an implementation with a different interface. Also called a wrapper.

Clojure analogue

• protocols - Any Clojure type can be extended to support a particular protocol using extend-type.

Keep in mind

• Using an adapter can be a symptom of a leaky abstraction.

Sample code

;; Shamelessly using protocols example from last post

;; our expect interface
(defprotocol MoneySafe
  "A type to convert other money type to bigdec"
  (make-safe [this] "Coerce a type to be safe for money arithmetic"))

;; our conversions, could also extend Java objects here.
(extend-protocol MoneySafe
  java.lang.Number 
  (make-safe [this] (bigdec this))

  java.lang.String
  (make-safe [this]
    (try
      (-> this
          (Double/parseDouble)
          (bigdec))
      (catch NumberFormatException nfe
        (println "String must be string of number characters"))
      (catch Exception e
        (println "Unknown error converts from string to money")
        (throw e))))

  clojure.lang.PersistentVector
  (make-safe [this]
    (try
      (let [num-bytes (->> this (filter int?) (count))]
        (if (= (count this) num-bytes)
          (->> this
               (map char)
               (clojure.string/join)
               (Double/parseDouble)
               (bigdec))
          (throw (ex-info "Can only convert from vector of bytes"
                          {:input this}))))
      (catch NumberFormatException nfe
        (println "Vector must be bytes representing ASCII number chars")
        (throw nfe))
      (catch Exception e
        (println "Error converting from vector of bytes to money")
        (throw e)))))

;; our client
(defn ->money
  [x]
  (make-safe x))

Template Method

Define the skeleton of an algorithm in an operation, deferring some steps to subclasses. Template Method lets subclasses redefine steps of an algorithm without changing the algorithm's structure.

Yet lots of people need template methods (functions?) since I've seen plenty of template functions in Clojure codebases over the years. Template functions get pretty hairy when the delegate functions get spread out over the codebase, doubly so if unamed lambdas are used. If you need to write functions across the vast expanse of your namespaces, use a strategy instead.

When to use it

Use a template method when you want to decouple primitives or subroutines of an algorithm from the overaching structure.

Clojure analogue

• Variodic function arguments with function composition - We could call this a template function rather than a template method. Variodic functions will give us the ability to define defaults as well as mandatory implementations.
• do-template - do-template evaluates an expression, grouping and feeding it's arguments into the template. It's good for a "template method" where the overrides are primitives or all "subtypes" of the template method/function are evaluated in the same client.

Keep in mind

• Template methods are very similar to the strategy pattern, but tightly couple the subtypes to the template, implying function definition locality is your friend.
• Depending on how important isolation of your variodic arguments is, You can swap out traditional variodic arguments for variodic keyword arguments in Clojure 1.11. For example, in our sample code below if we want to change the implementation of build-windows without changing build-foundation:

Sample code

;; our template defaults
(defn- build-foundation-default
  []
  (println "Building foundation with cement iron rods and sand"))
  
(defn- build-windows-default
  []
  (println "Building glass windows"))

;; The "template"
(defn build-house-template
  ([build-pillars build-walls]
   (build-house-template build-pillars
                         build-walls
                         build-foundation-default
                         build-windows-default))
  ([build-pillars build-walls build-foundation]
   (build-house-template build-pillars
                         build-walls
                         build-foundation
                         build-windows-default))
  ([build-pillars build-walls build-foundation build-windows]
   (build-foundation)
   (build-pillars)
   (build-walls)
   (build-windows)
   (println "House is built.")))

(defn wooden-house
  []
  (build-house-template #(println "Building pillars with wood coating")
                        #(println "Building wooden walls")))

(defn glass-house
  []
  (build-house-template #(println "Building pillars with glass coating")
                        #(println "Building glass walls")))

;; our client
(defn house-builder
  [build-house]
  (build-house))

Flyweight

Use sharing to support large numbers of fine-grained objects efficiently.

Naturally, the Strategy pattern would follow from the template function, but I decided to do flyweight next. Flyweight is basically a cache. If you want to get fancy, you can use the Clojure core.cache library where you get a cache protocol, conveinence macros, and eviction algorithms. Real caching. If you need to share some good ol' intrinsic state, here is a couple flyweight ideas for you.

When to use it

Use a Flyweight when you have overlapping object usage between clients. Flyweights typically destinguish between extrinsic state and intrinsic state where the extrinsic state is context dependend and intrinsic is context independent. So, the intrinsic state gets shared between client contexts.

Clojure analogue

• memoize - memoize will store the return value of a function mapped to it's arguments, conflating the two parts of the flyweight, the flyweight factory and operation, into one single memoize call.
• fn over an atom - We might not necessarily want to use our extrinsic state as our factory function key. In normal dev speak, that means we might not necessarily want our function arguments to determine what state gets shared between clients.

Keep in mind

• Introducing state (not the pattern) into your application means you have the potential to introduce certain bugs you wouldn't otherwise have. Use a watch to keep track of state changes if you're concerned about it.
• Using the atom version of the ->flyweight can be use in a local binding, but it will create a new atom, so be mindful of that.

Sample code

(defprotocol Sprite
  "Describes a totally real sprite type"
  (draw! [this x y height width]
    "draw something to the screen."))

(deftype PlayerSprite [sprite-sheet x y height width]
  Sprite
  (draw! [this x y height width]
    "Drew the player on the screen."))

(deftype EnemySprite [sprite-sheet x y height width]
  Sprite
  (draw! [this x y height width]
    "Drew the enemy on the screen."))

;; Our "flyweights"
;; could store anything in memoize.
(def player-sprite
  (memoize #(->PlayerSprite (slurp "./spritesheet.png")
                            {:x 0 :y 0 :height 32 :width 32})))

(def enemy-sprite
  (memoize #(->EnemySprite (slurp "./spritesheet.png")
                           {:x 33 :y 33 :height 32 :width 32})))


;; Atom example
;; our flyweight function is similar to the body of memoize
;; except we want to use a keyword to get value instead
(defn ->flyweight
  []
  (let [mem (atom {})
        player-coords {:x 0 :y 0 :height 32 :width 32}
        enemy-coords {:x 33 :y 33 :height 32 :width 32}
        sprites {:player #(->PlayerSprite (slurp "./spritesheet.png")
                                          player-coords)
                 :enemy #(->EnemySprite (slurp "./spritesheet.png")
                                        enemy-coords)}]
    (fn [k & args]
      (if-let [e (find @mem k)]
        (val e)
        (when-let [f (get sprites k)]
          (let [ret (apply f args)]
            (swap! mem assoc k ret)
            ret))))))

(def flyweight (->flyweight))

Strategy

Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

There's no official Clojure library for Strategy though. This is kind of a boring pattern in Clojure because it's just a lambda, so I'm including it completeness, but what blog post doesn't have fluff.

When to use it

Use a Strategy when you need to decouple algorithms from where they're being used.

Clojure analogue

• fn (again) - This time you'll have think real hard about the input and output of the higher-order functions you're passing around. sort-by allows you to pass in a key-fn as a strategy for getting the values to pass to the compare operation (or would that be a template method? Hmm....).

Keep in mind

• The usual stuff with fn. Name your fn for debugging and recursion. I suppose you could name them blah-blah-strategy, but you might get flak for saying the pattern name in the code.

Sample code

;; I'm not going to type out the sorts because I haven't
;; been in a CS classroom for years.
(defn quick-sort
  [coll])

(defn merge-sort
  [coll])

(defn radix-sort
  [coll])

(defn find-key
  "Performs a binary search on coll after calling
  sort-fn on coll."
  [key coll sort-fn]
  (find (sort-fn coll) key))
  
;; client somewhere
(find-key :fun-key (get-list) radix-sort)

1. Some people call this ports and adapters architecture.
2.The template method example was inspired by this Digital Ocean blog post.

Patterns

• Preamble
• Protocols
• Null Object
• Singleton
• Command
• Observer
• State
• Visitor
• Iterator

A motivational story

Suzi, after busting her ass at her first Clojure job, lands her second Clojure job at StartupAI. This company moved her half-way across the country to the Big City™, filling her with naive excitement. When she finishes company on-boarding, Suzi submits her first pull request after several days of staggering through the code. While the code is the same Clojure she has always known and loved, the repository gives her stress headaches.

The persistence namespace, the one with all the database queries, has eight different functions to get the same data using same query. Next, a minuscule change in the application layer causes dozens unit tests to fail, and to top it off, it takes three weeks to a pull request to get approved for the change.

"I moved across the country for this?!".

Here we go again

Although I'm sure some people will say it is, Suzi's dilemma isn't unique. Large Clojure codebases sometimes suffer from a lack proper care and abstraction. I suspect a lot of people rely too heavily on the pure function. Functions are great, but over time they create a codebase laden with high coupling and low cohesion. So, how do we help Suzi create a codebase satisfying our desired "ilities"? Lucky for Suzi, Clojure either directly supports, or easily supports design patterns like singleton, command, observer, visitor, and iterator. Before we get into those patterns though, we need to get little more abstract than the pure function.

We're speaking the same language

Let's start with the abstraction just above the function: Protocols. Protocols are Clojure's analogue to the interface. At a higher level, we can call a bag of function inputs and their outputs (or function signatures) an interface. Without diving into full-blown type theory, when something satisfies the interface, we can also say that something is a type of that interface. I'll even argue that we don't need a formal programming construct to achieve this "typehood". On a basic level, by definition, every Clojure sequence supports the functions cons ,first ,rest , and  count. Does that mean we should go out and create our own collection types? Not likely, the built-in types are quite good, but we do have few handy constructs for creating or extending types if we want.

(defprotocol MoneySafe
  "A type to convert other money type to bigdec"
  (make-safe [this] "Coerce a type to be safe for money arithmetic"))

(extend-protocol MoneySafe
  java.lang.Number
  (make-safe [this] (bigdec this))

  java.lang.String
  (make-safe [this]
    (try
      (-> this
          (Double/parseDouble)
          (bigdec))
      (catch NumberFormatException nfe
        (println "String must be string of number characters"))
      (catch Exception e
        (println "Unknown error converts from string to money")
        (throw e))))

  clojure.lang.PersistentVector
  (make-safe [this]
    (try
      (let [num-bytes (->> this (filter int?) (count))]
        (if (= (count this) num-bytes)
          (->> this
               (map char)
               (clojure.string/join)
               (Double/parseDouble)
               (bigdec))
          (throw (ex-info "Can only convert from vector of bytes"
                          {:input this}))))
      (catch NumberFormatException nfe
        (println "Vector must be bytes representing ASCII number chars")
        (throw nfe))
      (catch Exception e
        (println "Error converting from vector of bytes to money")
        (throw e)))))

(defn ->money
  [x]
  (make-safe x))
  
;; (+ (->money "0.1") (->money 0.1) (->money [0x30 0x2E 0x31])) => 0.3M

While we could define types with functions like sequence, we would lose the biggest benefit of the protocol construct, Polymorphism.

Null Object

Create an empty implementation for an interface.

By my "interface is a bag of functions" reasoning, nil makes a good Null Object for sequences, collections, and maps (or maybenil is a sequence, collection, and map, but I'll leave that to the Philosophers of Computer Science ;).

When to use it

When simulating the behaviour of a type without affecting the process.

Clojure analogue

nil - Clojure sequence, collection, and map operations all support nil as their input sequence, collection, and map, respectively, either returning an empty sequence, collection, map, or nil.

Keep in mind

Not everything supports nil, and nil is different than Java's null pointer reference, yet some gotchas with nil can throw NullPointerExecption, so some defensive programming with if or some->/some->> might be needed to avoid pitfalls.

Sample Code

Repl output:

user> (:keyword nil)
;; nil
user> (get nil :hi)
;; nil
user> (assoc nil :hi "bye")
;; {:hi "bye"}
user> (first nil)
;; nil
user> (rest nil)
;; ()
user> (next nil)
;; nil
user> (count nil)
;; 0
user> (nth nil 10)
;; nil
user> (pop nil)
;; nil
user> (peek nil)
;; nil
user> (println "You get the idea.")
You get the idea.
;; nil

Singleton

Ensure a class has one instance, and provide a global point of access to it.

A less philosophical pattern, the singleton appears a lot in Clojure codebases. Thanks to software transactional memory, we don't have to worry about non-atomic operations, but it's still an anti-pattern because one piece of client code may update the atom with something another piece of client code doesn't expect, causing a hard-to-debug error.

When to use it

Just don't.

Clojure analogue

defonce macro and an atom/agent

Keep in mind

Global state is best to be avoided unless absolutely necessary. Global configuration, database connection, and web servers are arguably important enough objects to use this, since they're the stateful sole instance and must stick around for the process runtime, but most Clojurians go in for a state management library or framework like Component, Mount, or Integrant.

Sample Code

(defonce web-server (atom nil))

(defn start-server
  []
  (if @web-server
    (log/warn "Server is already running!")
    (reset! webserver (run-jetty routes {:port (parse-int (:port config))
                                         :join? false}))))

Command

Encapsulate a request as an object, thereby letting users parameterize clients with different requests, queue or log requests, and support undoable operations.

The errors from functions are a lot easier to debug though because of stack traces. Our command pattern, the object-oriented version of callbacks, can be implemented in Clojure via higher-order functions (obviously).

When to use it

When you need to decouple the execution of a block of code from when it needs to be called.

Clojure analogue

Higher-order functions. All functions in Clojure can be passed around as arguments, and support closures.

Keep in mind

If you're passing around lambdas a lot, use the fn form and give it a name to help with debugging. The road to callback hell is paved with #(...) lambdas.

State with Callbacks are typically considered an anti-pattern, but JavaScript promises are a thing, so pick your poison.

Sample Code

;; Trivial example
(defn fire
  "fire!!!"
  []
  (println "Fire the gun"))

(defn jump
  "jump!!!"
  []
  (println "Jump"))

(defn run
  "run!#!#"
  []
  (println "Run enabled"))

(defn diagonal-aim
  "Shoot things on the walls"
  []
  (println "Aiming diagionally"))

(defn remap-button
  "Remap a player's button to a function.
  Can be used on the remap menu screen, or
  for temporary game mechanics."
  [player button f]
  (swap! player assoc-in [:controls button] f))

;; (remap-button player :a-button fire)

(defn do-buttons!
  "Execute the button actions assigned to the
  players controller map."
  [button-pressed controllor-map]
  (let [{:keys [a-button
                b-button
                y-button
                x-button]} controller-map]
    (case button-pressed
      "A" (a-button) ;; "Execute" the command!
      "B" (b-button)
      "X" (x-button)
      "Y" (y-button))))

Doing, redoing, and undoing

Because I know people will say it, here is an example of the command pattern supporting undo with all the previously made edits. Supporting redo will be left as an exercise for the reader.

(defn do-undo
  "Create a command and undo over a pure function"
  [f]
  (let [state (atom nil)]
    {:exec (fn execute!
             [& args]
             (let [v (apply f args)]
               (swap! state conj v)
               v))
     :undo (fn undo!
             []
             (swap! state pop)
             (peek @state))}))

(defn change-cell
  "A supposedly pure function"
  [s coord]
  (assoc sheet coord s))

(let [{:keys [exec undo]} (do-undo change-cell)]
  (def change-cell-cmd exec)
  (def change-cell-undo undo))

Observer

Define a one-to-many dependency between objects so that when one object changes state, all the dependents are notified and updated automatically.

I see a lot of functions written to watch for something to happen. Typically, developers will use a loop over a function until some state changes or "event" transpires. This works, but couples the event to the event reaction. Object-oriented programming addresses this with the Observer pattern, but we can address it with a Clojure watch.

When to use it

When you want to decouple changes in state from what the state affects, also known as a publish/subscribe pattern.

Clojure analogue

There are two ways to do this in Clojure:

• Clojure watches. You can add a watch to any atom, agent, var, or ref with add-watch and remove it with remove-watch.
• core.async's pub and sub. pub creates a channel from a existing channel with optional topic, and then other channels can sub to the pub channel.

Keep in mind

When the watch target (the var/atom/agent/ref) changes, all the watch functions will be called syncronously even if multiple threads are manipulating the watch target. So, it's best to use the new-state argument in all your watch functions.

Sample Code

(defn unlock
  "Unlocks the achiement given"
  [player k]
  (when-not (unlocked? player k)
      (do-unlock! player k)))

(defn on-bridge?
  "predicate to see if player is on the bridge"
  [player]
  (is-colliding? (get @entities :bridge) @player))

(defn achievement-toaster
  [k]
  (let [a (get achievements :player-fell-achievement)]
      (when a
        (swap! entities :current-toaster a))))

(defn unlock-achievement
  [key player old-state new-state]
  (case (:last-event new-state)
    :player-fell (when (on-bridge? new-state)
                   (unlock player :player-fell-achievement)
                   (achievement-toaster :player-fell-achievement))))

(defn watch-actions
  [player]
  (add-watch player :achievements unlock-achievement))

State

Allow an object to alter its behavior when its internal state changes. The object will appear to change it's class.

We might not care about reacting to the state change though. Heck, we might not even care how the state happens, we just care about the results. A state pattern can give use a nice function to conceal our messy state transitions, so we don't have to write giant cond statements.

When to use it

The real thing to remember here is that we want to model a finite state machine (FSM) with our functions.

Clojure analogue

• (atom f) - An atom over a function is the simplist version of the state pattern.
• (loop [state (init-state data)] (recur (state data))) - Alternatively, the the current state could be re-bound a loop binding since the current state function always returns the next state.

This take uses a multimethod over an atom. This certainly models an FSM, but we kind of lose our ability to decouple the state from the context.

Keep in mind

• The current state will always return the next state in the machine. Modeling state this way can be tricky, but can be super helpful if you find yourself with many nested case or cond forms. You'll have to decide of the cyclomatic complexity is worth the trade-off.

• When your state transition returns one state, this method is fine. If you need more functions in a single 'state', I'd consider using protocols to create states. I intentionally left this out of the sample code because it was huge.

• If you need a history for your state machine, like a push-down automata, you can conj each state function to a vector and peek at the top.

• The atom-free state pattern with loop bindings can be problematic if the current state needs to modify the binding itself. Use the atom version instead.

Sample Code

(defn play-game
  [game]
  (let [{:keys [components player]} game]
    (do
      (doseq [component components]
        (update! component))
      (draw-game game)
      (if (= :paused (:action @player))
        pause-menu
        play-menu))))

(defn start-menu
  [game]
  (let [{:keys [menu player]} game]
    (do
      (draw-game game)
      (if (= :affirm (:action @player))
        (when-let [menu-item (collision player)]
          (cond ;; our state transition
            (= menu-item :start-game) play-game
            (= menu-item :options) options-menu
            (= menu-item :quit-game) quit-game
            :else start-menu))))))

(let [game-state (atom start-menu)]
  (defn start-game
    [game]
    (reset! game-state (@game-state game))))
    
(defn main
  []
  (let [game-engine (game-engine)]
    (loop [quit? (:quit game-engine)]
      (when-not quit?
        (start-game game-engine)
        (recur (:quit game-engine))))))
                   
;; or stateless 'state'
(defn stateless-start-game
  [game]
  (start-menu game))

(defn stateless-main
  []
  (let [game-engine (game-engine)]
    (loop [game-state (start-game game-engine)]
      (when-not (:quit game-engine)
        (recur (game-state game-engine))))))

Visitor

Represent an operation to be performed on elements of an object structure. Visitor lets you define a new operation without changing the classes of the elements on which it operates.

Giant case statements are sometimes addressed with multimethods. That's a valid strategy, but multimethods give us multiple dispatch for free.

When to use it

When you want to create new functionality without changing the existing code structure.

Clojure analogue

multimethods. Clojure's multimethods offer dynamic dispatch functionality right out of the box, so we can put a multimethod inside of a multimethod to get the double dispatch functionality.

Keep in mind

Multimethods have a performance penalty. You're unlikely to experience it, but it is there, so just be mindful of it in performance sensitive applications. If you need fast dispatching, consider dispatch on protocols instead.

Sample Code

(defrecord Equipment [elements])
(defrecord Sword [name])
(defrecord QuestItem [name])
(defrecord Armor [name])

(defmulti visit (fn [f x y] [(class x) (class y)]))

(defmethod visit :default [f x y])

(defmethod visit [Equipment Object] [f equipment other]
  (let [{:keys [elements]} equipment]
    (doseq [element elements]
      (f element other))))

(defmulti do-stuff (fn [x y] [(class x) (class y)]))

(defmethod do-stuff :default [x y]
  (println (format "You sit there bewildered, unable to understand what to do with %s and %s" x y)))

(defmethod do-stuff [Sword Long] [x y]
  (println (format "Swing your sword %s does %d damage" (:name x) y)))

(defmethod do-stuff [Sword String] [x y]
  (println (format "Using your sword %s, you make a noble gesture %s" (:name x) y)))

(defmethod do-stuff [Armor Long] [x y]
  (println (format "Your armor %s has prevented %d damage" (:name x) y)))

(defmethod do-stuff [Armor String] [x y]
  (println (format "You remove armor %s as a noble gesture %s" (:name x) y)))

(defmulti store-stuff (fn [x _] (class x)))

(defmethod store-stuff :default [x y]
  (println "You stored: " x))

Repl output:

user> (def equipment (->Equipment [(->Sword "Excalibur") (->Sword "Gramr") (->Sword "Zulfiqar") (->Sword "Durendal") (->QuestItem "dungeon key") (->Armor "Achilles")]))
;; #user/equipment
user> (visit store-stuff equipment 10)
You stored:  #user.Sword{:name Excalibur}
You stored:  #user.Sword{:name Gramr}
You stored:  #user.Sword{:name Zulfiqar}
You stored:  #user.Sword{:name Durendal}
You stored:  #user.QuestItem{:name dungeon key}
You stored:  #user.Armor{:name Achilles}
;; nil
user> (visit store-stuff equipment nil)
;; nil
user> (visit do-stuff equipment 50)
Swing your sword Excalibur does 50 damage
Swing your sword Gramr does 50 damage
Swing your sword Zulfiqar does 50 damage
Swing your sword Durendal does 50 damage
You sit there bewildered, unable to understand what to do with user.QuestItem@bb0c2cb0 and 50
Your armor Achilles has prevented 50 damage
;; nil

Iterator

Provide a way to access the elements of an aggregate object sequentially without exposing its underlying representation.

Like our Visitor pattern, the Iterator doesn't really reveal anything special. We can use it without any extra effort which makes this section fluff. Enjoy!

When to use it

When you want to have an ordered, sequential container for other objects which sounds an awful lot like a sequence or collection.

Clojure analogue

Lists and Vectors - Clojure list and vector types are sequence and collection, respectively, and support arbitrary typed elements. They have a rich set of functions for operating on them like doseq for map filter etc.

Keep in mind

Make sure you use doseq for side-effects. Some sequences (and transforms from collections) are Chunked, and can create strange behavior if using side-effects with map or filter.

A Java Iterator (supporting the java.util.Iterator interface) is not a sequence in Clojure. If you are working with a Java Iterators, be sure to call the iterator-seq function on your Java Iterator to get a (chunked) sequence. Some Java types do return collections though, and those should be fine in map or filter since Clojure collections inherit from the Java collections.

Sample code

(->> (range 1000) ;; Range gives us a list of numbers
     (filter even?) ;; Plugs into 
     (map inc))

1. Because it's my favorite book on this subject, the inspiration for this post by and large came from Game Programming Patterns (print). The sample code for Command and Observer, as well as each pattern's section structure, was adapted for this post's theme.
2. This post is also largely a rebuttle to mishadoff's Clojure design patterns.
3. Except for the Null Object, all quotes underneath pattern names come directly from Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides (1994). Design Patterns: Elements of Reusable Object-Oriented Software. Addison Wesley. ISBN 0-201-63361-2.

Suzi, a software engineer of suitable caliber, loves writing Clojure. At her polyglot employer, she often uses her technical freedom to bat for Clojure every chance she gets. One day her manager taps her for an opportunity, the company wants to build a scrappy, lean internal tooling team to build tools for sales, marketing, and so on.  Perfect.

One the first day of the tooling team, the marketing team wants to move the website to something custom and use Stripe for payments from the lower segment customers. No problem. Clojure's web stack is very mature and will be sinch to whip-up this project for marketing. The website was easy enough to tackle, a little ring here, a little hiccup there, and presto! She has her website after just a couple of weeks of coding.

Next on her docket was the Stripe payment integration. Should be the same thing as the website. Suzi sees there are a number of Stripe Clojure libraries to choose from.

"Perfect" Suzi thinks to herself as she were in developer heaven.

Upon her first test with the repl, she gets an exception. She inspects her Stripe library by looking at the master branch on Github: Last commit 7 years ago. F**k. It's just some simple requests with clj-http and cheshire. Next library. Last commit 3 years ago.

"Okay, looking better" Suzi thinks to herself as she lets out a sigh of relief.

This library is almost the same thing as the last one, simple and only covering a fraction the API. Unsure if this new library actually keeps with the current API contract, no support for lazy pagination like the Java SDK, and it suffers from the Log4j vulnerability. Fantastic, so much for that estimate. The next day during stand up, Suzi explains to her colleagues that the project is going to take a bit longer because she didn't anticipate having to update libraries and implement features the Java crowd gets for free.

Fast forward a couple months. After the marketing website hurdles, a new project from IT is requested. They want a syslog relay for SOC2 monitoring stuff. That doesn't really sound like SOC2 compliance to Suzi, but she has her Clojure hammer and a good salary, so she gets to work. Once she finishes the requirements, she begins researching to give an estimate of how long the project will take, only to find there is no syslog server library for Clojure that she can find. She will have to write one herself. During stand up the next day, Suzi delivers the estimate to her team with her line manager, Sam, present.

"It'll probably take two months to build this relay because there's no syslog server library for Clojure." she says.

"Oh wow, I didn't think it would take that long." Sam says.

Dillion, a colleague in the standup, says "Go has great syslog server library and even supports Rsyslog! Maybe you could use that".

"That's probably a good idea, Suzi can estimate how long it'll take using Go?" Sam replies.

Later that day, Suzi estimates the project with Go instead. The business case for Go is very strong now. Sh*t.

Strange yet familiar

If Suzi's story resonated with you, you're like me and dozens of other Clojure developers. Clojure suffers from missing or rotting software libraries a lot. Developers love writing code, but it's really hard to sell management on building an entire library when another language has so many batteries built in.

In a perfect world of open-source software, companies would give back to the community by building libraries they need and licensing them in the wild. While some software companies do this, in my experience, it's usually the opposite that happens, most companies don't give back, but I think there's another reason this problem persists in Clojure. There's no Rails for Clojure.

Railing against Rails

Because there is no Rails for Clojure, a lot of open-source software efforts in the Clojure community go towards developing web frameworks, libraries, toolkits, and so on, trying to fill this "Rails for Clojure" vacuum. Yet, the web stack for Clojure is very mature.

- Ring (server)
- Compojure (router)
- Edge (framework)
- Bidi (router)
- Yada (library, bidi + aleph)
- Aleph (server)
- Reitit (router)
- Luminus (many libraries)
- Kit (framework)
- Pedestal (many libraries)
- Selmer (templates)
- Hiccup (HTML DSL)

Every time I see Clojure on the front page of Hacker News, it's a post talking about a new web framework, library, toolkit, or abstraction for Clojure, and (usually) the maintainer has abandoned the project since revealing it to the world. My theory is, this causes a lack of effort expended on mundane libraries. It's not the most fun thing to build. I get it. Really, I do, but lack of these API-wrapping and protocol-handling libraries put Clojure at a significant disadvantage in the business of software.

Interop for the mundane

Why not just use Java and Javascript interop for the mundane, everyday libraries, is there something wrong with that?

It's possible, but interop comes with caveats.

• Interop can be frustrating with big frameworks.
• Interop relies heavily on reflection, and may require type hinting once it becomes a performance problem.
• Interop implies the use of side-effects since we're relying on Java objects.
• Java sequences and Collections may require a clojure.walk/postwalk and zipmap with coercion to get a good Clojure container/sequence/collection.
• No variodic arguments to deftype or defprotocol for technical reasons, meaning overloading isn't happening when extending Java types.

If we used interop for everything mundane, Clojure would really just be an S-expression shaped husk over Java code. Not a very good solution.

Rallying on the tracks

Other programming languages have their definitive web framework. Ruby has Rails, Python has Django, Java has Play, Elixir has Pheonix. I know there are other web framework, libraries, toolkit, and so on for each of these programming languages, but their communities always seems to rally around each of these projects. Even a language as niche as Elixir, has it's own "Rails" (so to speak).  

Why doesn't Clojure do this? From my observation, the Clojure community seems a bit more de-centralized and laid back. It's awesome, but can we please just get together at the next Conj conference decide what our "Rails" is going to be? That way we can get down to business.