Clojure 1.2 has been released and Leiningen has moved up to 1.3.0 (or was it briefly 1.2.0) which supports the latest version - so I decided to spend a spare afternoon playing with one of the new features of Clojure.

One of the interesting new features is datatypes which are a replacement for the structmap feature from earlier versions of Clojure. Datatypes and structmaps are directly equivalent to maps. Clojure provides an implementation of relational algebra that works on its maps in the clojure.set namespace, I expected that these functions should work with datatypes but I decided to check for myself.

Read more…

A little while back, I was informed that the V&A had an API. To be honest, my first response to this was “why on earth?”. There’s been a few similar APIs coming out recently from organisations, with some sort of “build it and they’ll come” expectations i.e. expecting that all they have to do is provide the API and all us developers will automagically build them shiny apps for free. If you’re TfL, then this kinda works, but it’s not so true for a lot of places.

Having had this initial reaction, I still decided to dig through the documentation a bit, and spotted an interesting nugget - they’ll let you do geospatial searches. I’d been tinkering around with the idea of playing with this, especially for use with my shiny new Android phone, and I had an a idea for a little app to show you “nearby art” i.e. search with the V&A’s API for the nearest bit of art.

I did this mostly in Javascript, doing XMLHttpRequest’s for JSON chunks of the API. There’s also a block of Python code that needs to run on a server, but that’s entirely to get around the issues of XMLHttpRequest only allowing same-server requests. It first uses navigator.geoLocation (official spec, easier documentation) to get the user’s location, then does two V&A queries - the first to get a list of local objects, and the second to get more info on the first object.

One thing you have to be careful about is that this can break in various ways. The most obvious is a lack of navigator.geoLocation (any version of IE, and all not-latest versions of most other browsers), and another is if the user denies access to their location data. This does make navigator.geoLocation unsuitable for general use currently, but it’s a useful source of data when there is support.

The full app is over here and the source is here.

Previously (here) I sketched out a design that went some way to being a coverage tool for Clojure code. The result was a macro that when used to define a function, resulted in an instrumented function that tallied every time the code was called. This time I am going to sketch the beginnings of a design that doesn’t use a macro.

To start with here is a slight re-write of the structure I used last time to record the coverage:

(def coverage-records (ref nil))

(defn add-record [fn-name record]
  (dosync (alter coverage-records assoc (str fn-name) record)))

(defn get-record [fn-name]
  (get @coverage-records (str fn-name)))

(defn covering [form fn-name]
  (dosync (alter (get-record fn-name) assoc form 0)))

(defn- inc-map [map key]
  (if (contains? map key)
    (assoc (dissoc map key) key (inc (get map key)))
    map))

(defn inc-coverage [form fn-name]
  (dosync (alter (get-record fn-name) inc-map form)))

This code uses a single map to store a coverage record, another map, for each function that is being measured. Functions are provided for adding records and incrementing the coverage.

I am using the same set of functions to wrap and instrument the s-expressions that define the function, with a few minor modifications. I am indexing the s-expressions with a number so as to distinguish between any identical s-expressions within the function. This set of functions is still incomplete since there are lots of Clojure code that will get wrapped incorrectly, I will be rectifying this sometime in the near future.

(declare wrap-seq)
(defn wrap
  [form idx fn-name]
  (cond
    (seq? form) (
      let [key (str idx ":" form)]
        (covering key fn-name)
        (list ‘do `(inc-coverage ~key ~(str fn-name)) (wrap-seq form idx fn-name)))
    :else form))

(defn- indexed [coll] (map vector (iterate inc 0) coll))

(defn wrap-seq [coll count fn-name]
  (for [[idx elt] (indexed coll)] (wrap elt (+ count idx) fn-name)))

The previous article used a macro to generate Clojure code that was instrumented, this time I will load the source code for the function, wrap it and then evaluate it. This is done by the get-source function in clojure.contrib.repl-utils combined with a call to format and load-string, My code that interprets the source code is fragile and needs more work, it won’t handle functions with multiple bodies for example, but it demonstrates the principle. I am using a structure to hold the instrumented function and the coverage record for this function.

(defstruct wrapper :wrapped-fn :coverage-record)

(defn wrap-fn
  [f cv-rec]
  (let [s (read-string (get-source f))
        fn-name (first (drop 1 s))
        args (first (drop 2 s))
        body (last s)]
    (add-record fn-name cv-rec)
    (load-string (format “(fn %s %s)” args (wrap body 0 fn-name)))))

(defn wrap-function [f]
  (let [coverage-record (ref nil)]
    (struct wrapper (wrap-fn f coverage-record) coverage-record)))

So now we can try the code out at the REPL:

user=> (def x (wrap-function 'test1))   
#'user/x
user=> x
{:wrapped-fn #, :coverage-record #}

Here I have wrapped the function test1 and we can see the coverage structure returned, consisting of the wrapped function and the coverage record, keyed by the single s-expression with a count of 0. If I now use the wrapped function like this and examine the coverage structure the count should increment to 1.

user=> ((:wrapped-fn x) 1 2)            
-1
user=> x
{:wrapped-fn #, :coverage-record #}

So we now have a partially functional coverage tool and no macros have needed to be written. To complete this tool I just need to tidy up the wrapping and source code reading functions and provide some sort of binding macro so that we can call the fn with

(test1 1 2)

((:wrapped-fn x) 1 2)

One of the internal tools that some of us use is a little script called getcap. Using the venerable xwd and ImageMagick, along with a bit of shell scripting it gets run by cron 8 times an hour and dumps a screen capture into a hidden folder in your home directory. As you can imagine, this is the sort of thing that if someone else was doing it would be a nasty invasion of privacy, but if you’re doing it for yourself, it becomes invaluable for accurately filling in timesheets, especially when you’re busy juggling 3-4 different projects.

I’d run into a few problems with it however, primary being that xwd doesn’t play nice with compositing, and so my Docky instance at the top of the screen looks like a big black patch across the screenshots, which occasionally blocks useful information.

I’ve therefore written a replacement, called LittleBrother, using primarily Python and gtk. Initially I was just following the instructions on the PyGtk FAQ on how to capture your screen, but then I figured I could do a bit more. So, we now have a little status bar icon that lets you open up the screenshots folder (using xdg-open so the right file manager gets used), and instead of having to hardcode in the times when you’re in the office, it instead uses the DBus interface of the Gnome screensaver (patches for other screensavers welcomed) to detect when the screensaver is active, and only take captures when it’s inactive. I found the D-feet DBus introspection tool particularly useful for playing around with this step.

Code is available on Github, and there’s even a Debian package available as well.

As part of a customer project some years ago, we wrote an implementation of the interesting parts of RFC 3339 for Python. The abstract for the RFC says

This document defines a date and time format for use in Internet protocols that is a profile of the ISO 8601 standard for representation of dates and times using the Gregorian calendar.

We needed to be able to robustly transfer timestamps between languages (Javascript and Python, chiefly) without getting tangled up in timezone troubles or complex ambiguous parsing problems.

Our code provides

• simple, standard, robust, cross-language (e.g. Javascript) format for parsing and printing time stamps
• a standard no-frills “UTC” tzinfo class and singleton instance
• a standard no-frills fixed-offset tzinfo class
• other utilities for helping write robust timezone-aware time manipulation code

Examples

These examples are taken from the doctests/docstrings in the module source itself. See the module documentation for many more informative examples.

Parsing a timestamp, with timezone support and timestamp equivalence:

>>> midnightUTC = parse_datetime("2008-08-24T00:00:00Z")
>>> oneamBST = parse_datetime("2008-08-24T01:00:00+01:00")
>>> midnightUTC == oneamBST
True

Printing a timestamp:

>>> oneamBST.isoformat()
'2008-08-24T01:00:00+01:00'
>>> parse_datetime("2008-08-24T00:00:00.123Z").isoformat()
'2008-08-24T00:00:00.123000+00:00'

Downloading the code

The code is available on github. It’s MIT-licensed.

• Browse the code here
• Download a tarball of the latest code

You can also install the module directly from github using pip:

pip install -e git://github.com/tonyg/python-rfc3339.git#egg=rfc3339

In addition to our swing visualizer in Java, we now have something closer to a two-dimensional swingometer in JavaScript. Not quite as polished as I’d hoped this late, but still useful in an unpredictable election. I hope to be updating both on election night. Suggestions welcome!

Read more…

I like to write unit tests for my code and I also like to know whether my unit tests are actually testing my code. In Java I would use Maven and Cobertura to measure how much of my code is actually exercised, and when I bend my mind around Haskell I use HPC.

I have been experimenting with Clojure and it has a nice testing library but the only way I have found to measure coverage is to use Cobertura and squint at the generated class names to work out what I haven’t covered. So I have started on the road to creating a coverage tool for Clojure written in Clojure. This blog entry contains my initial steps and design sketches towards creating the tool.

Ideally I’d like to measure my coverage like this:

(ns some.tests
  (:use [some.functions])
  (:use [coverage.cover])
    (:use [clojure.test]))

(deftest test-something
    (cover ‘(foo bar)                   ; cover foo and bar
      (is (= [1 2 3] (foo 1 2 3)))
        (is (= [2 4 6] (bar 1 2 3)))

        (is (= [100 100] (coverage foo))) ; check coverage
        (is (= [100 100] (coverage bar)))

        (store-coverage)))                ; store report to file

That isn’t functional code yet and may not be what I finally produce, it might not even be valid Clojure since I haven’t even bashed it into a REPL yet.

As Clojure is a Lisp, and we are testing our own code - i.e. we have the source code - we should be able to measure the coverage using Clojure. This can achieved by rebinding the functions under test with instrumented functions; this is what I envisage the cover function is doing:

(cover '(foo)
  ; all calls to foo here actually call foo'
)

Which can be achieved using a binding function:

(binding [a b)]
    ; all calls to a here are actually calls to b
)

So assuming I can write the cover function I will need to instrument a function. Let’s start simply and ignore the complicated stuff such as conditional statements and multiple bodies until later. If we have this simple function

(defn foo [a b c] (+ a (- b c)))

I would like an instrumented version to wrap each s-expression with a function that records that the s-expression has been called, for now we will just print a message.

(defn foo-wrapped [a b c]
  (do
      (prn :wrap)
        (+ a
          (do
            (prn :wrap)
              (- b c)))))

Testing that in the REPL seems to reveal that I am moving in the right direction:

user=> (foo 1 2 3)
0
user=> (foo-wrapped 1 2 3)
:wrap
:wrap
0
user=>

To convert the first function to the instrumented function I need a couple of auxiliary functions and a macro:

(defn record [] (prn :wrap))

(declare wrap-seq)
(defn wrap [form]
  (cond
      (seq? form)
          (list ‘do (record) (wrap-seq form))
        :else form))

(defn wrap-seq [xs]
  (for [x xs] (wrap x)))

(defmacro cover
  [args & body]
    (let [fn-name (when (symbol? args) args)
          args (if fn-name (first body) args)
                body (if fn-name (next body) body)]
      (fn ~@(if fn-name (list fn-name args) (list args))
          ~@(wrap (first body)))))

If you know some Clojure that code demonstrates some of the problems I haven’t solved yet when measuring coverage, for example the :else clause will need to be covered as well as functions in binding statements. Testing that code in the REPL gives us a reasonable result:

user=> (def x (cover [a b c d] (+ a (- b c (* a d)))))
#’user/x
user=> (x 1 2 3 4)
:wrap
:wrap
:wrap
-4
user=>

So if I now replace the record function with something else I may be able to generate some coverage for a function. I am going to use a map held by a ref to store the coverage information, a function to register a particular s-expression with the ref and a function to increment the count.

(def coverage-records (ref nil))

(defn covering [form]
  (dosync (alter coverage-records assoc form 0)))

(defn- inc-map [map key]
  (if (contains? map key)
    (assoc (dissoc map key) key (inc (get map key)))
    map))

(defn inc-coverage [key]
  (dosync (alter coverage-records inc-map key)))

Lets test that:

user=> @coverage-records
nil
user=> (covering :x)
{:x 0}
user=> (covering :y)
{:y 0, :x 0}
user=> (inc-coverage :x)
{:x 1, :y 0}
user=> (inc-coverage :y)
{:y 1, :x 1}
user=> (inc-coverage :x)
{:x 2, :y 1}
user=> @coverage-records
{:x 2, :y 1}
user=>

If I modify our wrapping functions I can create the coverage records ready to be incremented.

(defn wrap [form]
  (cond
    (seq? form) (do
      (covering (str form))
      (list ‘do `(record) (wrap-seq form)))
    :else form))

Testing that code:

user=> (def x (cover [a b c d] (+ a (- b c (* a d)))))
#’user/x
user=> @coverage-records
{”(* a d)” 0, “(- b c (* a d))” 0, “(+ a (- b c (* a d)))” 0}
user=>

And that is as far as I have got. I need to change my record function to increment the coverage records, make my wrap function cope with conditionals and other complexities and write my rebinding function. Hopefully next month I will have moved it along a little bit further.

SpringSource, a division of VMware, Inc. today announced the acquisition by VMware of Rabbit Technologies, Ltd, a company set up by LShift and partners Monadic and CohesiveFT.

Read the full story

When I’m doing development on an existing software project, and especially when I’m trying to bugfix something with a Debian package, I find that I install random packages I need to rebuild something, and then later on I’m wondering why I’ve got those installed. I tend to try to keep with the philosophy that the bits of software I’m using are pretty knowledgeable about whatever they’re intended to do, and so trusting them to make smart decisions is a good idea. For package managers, this means I should only keep track of the software that I actually use, and tell the package manager that everything else has been automatically installed and can therefore be removed when they cease to be a dependency of something I actually need. Having packages that I only installed for building something else without a record of that in the package management system breaks that mental model.

Enter dh-builddep-metapackage (the naming is inline with the names used by the debhelper scripts used for other debian packaging stuff). dh-builddep-metapackage builds build-dependency metapackages in order to ease package management for package rebuilders. In effect, it builds a “<package name>-builddep” package that has no content, but depends on everything that the existing package build-depends on. By using dh-builddep-metapackage to create metapackages rather than using “apt-get build-dep”, I keep a record in the package management about why I need a particular development package, and can remove the dependant packages when I’m no longer working with the relevant source package.

I think someone’s done this before, but some work with Google didn’t find it, and it was an interesting exercise anyway. Standard usage is “dh-builddep-metapackage -b <package name>”, which will create the metapackage data and build the package for you using dpkg-buildpackage. A folder called “<package name>-<package version>” will be created in the current local directory, and if an existing folder exists then dh-builddep-metapackage will refuse to overwrite it (unless you give the -o/–overwrite option).

So far I’ve used it a few times, and it’s quite nice to have those packages around to remind me about the dependencies. I’m considering building a full repository with *-builddep packages for everything in the Debian archives, but I’d like to make sure I’d use this enough first.