Design sketches for Clojure coverage tool

I like to write unit tests for my code and I also like to know whether my unit tests are actually testing my code. In Java I would use Maven and Cobertura to measure how much of my code is actually exercised, and when I bend my mind around Haskell I use HPC.

I have been experimenting with Clojure and it has a nice testing library but the only way I have found to measure coverage is to use Cobertura and squint at the generated class names to work out what I haven’t covered. So I have started on the road to creating a coverage tool for Clojure written in Clojure. This blog entry contains my initial steps and design sketches towards creating the tool.

Ideally I’d like to measure my coverage like this:

(ns some.tests
  (:use [some.functions])
  (:use [coverage.cover])
    (:use [clojure.test]))

(deftest test-something
    (cover ‘(foo bar)                   ; cover foo and bar
      (is (= [1 2 3] (foo 1 2 3)))
        (is (= [2 4 6] (bar 1 2 3)))

        (is (= [100 100] (coverage foo))) ; check coverage
        (is (= [100 100] (coverage bar)))

        (store-coverage)))                ; store report to file

That isn’t functional code yet and may not be what I finally produce, it might not even be valid Clojure since I haven’t even bashed it into a REPL yet.

As Clojure is a Lisp, and we are testing our own code - i.e. we have the source code - we should be able to measure the coverage using Clojure. This can achieved by rebinding the functions under test with instrumented functions; this is what I envisage the cover function is doing:

(cover '(foo)
  ; all calls to foo here actually call foo'
)

Which can be achieved using a binding function:

(binding [a b)]
    ; all calls to a here are actually calls to b
)

So assuming I can write the cover function I will need to instrument a function. Let’s start simply and ignore the complicated stuff such as conditional statements and multiple bodies until later. If we have this simple function

(defn foo [a b c] (+ a (- b c)))

I would like an instrumented version to wrap each s-expression with a function that records that the s-expression has been called, for now we will just print a message.

(defn foo-wrapped [a b c]
  (do
      (prn :wrap)
        (+ a
          (do
            (prn :wrap)
              (- b c)))))

Testing that in the REPL seems to reveal that I am moving in the right direction:

user=> (foo 1 2 3)
0
user=> (foo-wrapped 1 2 3)
:wrap
:wrap
0
user=>

To convert the first function to the instrumented function I need a couple of auxiliary functions and a macro:

(defn record [] (prn :wrap))

(declare wrap-seq)
(defn wrap [form]
  (cond
      (seq? form)
          (list ‘do (record) (wrap-seq form))
        :else form))

(defn wrap-seq [xs]
  (for [x xs] (wrap x)))

(defmacro cover
  [args & body]
    (let [fn-name (when (symbol? args) args)
          args (if fn-name (first body) args)
                body (if fn-name (next body) body)]
      (fn ~@(if fn-name (list fn-name args) (list args))
          ~@(wrap (first body)))))

If you know some Clojure that code demonstrates some of the problems I haven’t solved yet when measuring coverage, for example the :else clause will need to be covered as well as functions in binding statements. Testing that code in the REPL gives us a reasonable result:

user=> (def x (cover [a b c d] (+ a (- b c (* a d)))))
#’user/x
user=> (x 1 2 3 4)
:wrap
:wrap
:wrap
-4
user=>

So if I now replace the record function with something else I may be able to generate some coverage for a function. I am going to use a map held by a ref to store the coverage information, a function to register a particular s-expression with the ref and a function to increment the count.

(def coverage-records (ref nil))

(defn covering [form]
  (dosync (alter coverage-records assoc form 0)))

(defn- inc-map [map key]
  (if (contains? map key)
    (assoc (dissoc map key) key (inc (get map key)))
    map))

(defn inc-coverage [key]
  (dosync (alter coverage-records inc-map key)))

Lets test that:

user=> @coverage-records
nil
user=> (covering :x)
{:x 0}
user=> (covering :y)
{:y 0, :x 0}
user=> (inc-coverage :x)
{:x 1, :y 0}
user=> (inc-coverage :y)
{:y 1, :x 1}
user=> (inc-coverage :x)
{:x 2, :y 1}
user=> @coverage-records
{:x 2, :y 1}
user=>

If I modify our wrapping functions I can create the coverage records ready to be incremented.

(defn wrap [form]
  (cond
    (seq? form) (do
      (covering (str form))
      (list ‘do `(record) (wrap-seq form)))
    :else form))

Testing that code:

user=> (def x (cover [a b c d] (+ a (- b c (* a d)))))
#’user/x
user=> @coverage-records
{”(* a d)” 0, “(- b c (* a d))” 0, “(+ a (- b c (* a d)))” 0}
user=>

And that is as far as I have got. I need to change my record function to increment the coverage records, make my wrap function cope with conditionals and other complexities and write my rebinding function. Hopefully next month I will have moved it along a little bit further.