One of the interesting new features is datatypes which are a replacement for the structmap feature from earlier versions of Clojure. Datatypes and structmaps are directly equivalent to maps. Clojure provides an implementation of relational algebra that works on its maps in the clojure.set namespace, I expected that these functions should work with datatypes but I decided to check for myself.

I specifically wanted to check how close you an approach to an ideal relational system as defined by C. J Date in his books - Database in Depth and Databases, Types, and the Relational Model: The Third Manifesto. Date strongly believes that databases should be built on relational algebra and that none of the current relational databases achieve that goal. Database in Depth is a very readable and concise introduction to relational algebra and is well worth reading.

clojure.set contains six functions that correspond to the six fundamental operations in relational algebra - union, difference, rename, selection, projection and join. The first two, union and difference, are simple set operations that will definitely work on sets of datatypes. Let us try the other four operators on some datatypes, I’ll base my examples on a cut-down set of relations used by Date describing suppliers, parts and shipments.

A clojure datatype is created using the defrecord function. We will a datatype for a supplier, a part and a shipment:

(defrecord Supplier [number name status city])
(defrecord Part [number name colour weight city])
(defrecord Shipment [supplier part quantity])

We can create a supplier like this:

(Supplier. "S1" "Supplier 1" 100 "London")

We can then define our three relations like this:

(def suppliers
  #{(Supplier. "S1" "Smith" 20 "London")
    (Supplier. "S2" "Jones" 10 "Paris")
    (Supplier. "S3" "Blake" 30 "Paris")})
(def parts
  #{(Part. "P1" "Nut" "Red" 12.0 "London")
    (Part. "P2" "Bolt" "Green" 17.0 "Paris")
    (Part. "P3" "Screw" "Blue" 17.0 "Oslo")})
(def shipments
  #{(Shipment. "S1" "P1" 300)
    (Shipment. "S2" "P2" 200)
    (Shipment. "S3" "P3" 400)})

Now let us try the first relational operation rename:

clojure.core=> (use 'clojure.set)
nil
clojure.core=> (rename parts {:number :id :city :location})
 #{{:location "London", :name "Nut", :colour "Red", :weight 12.0, :id "P1"}
 {:location "Paris", :name "Bolt", :colour "Green", :weight 17.0, :id "P2"}
 {:location "Oslo", :name "Screw", :colour "Blue", :weight 17.0, :id "P3"}}

So this results in a new map based on the parts relation with the number and city fields renamed to id and location respectively.

Now let us try a select operation:

clojure.core=> (select #(= (:name %) "Smith") suppliers)
#{#:Supplier{:number "S1", :name "Smith", :status 20, :city "London"}}

Note in this case the set that has been returned contains the datatype defined earlier not a simple map.

Now let us try the project operation:

clojure.core=> (project suppliers [:city])
#{{:city “London”} {:city “Paris”}}

Finally let us join the parts and shipments relations:

clojure.core=> (join parts shipments {:number :part})
#{#:Part{:number "P1", :name "Nut", :colour "Red", :weight 12.0, :city "London", :quantity 300, :part "P1", :supplier "S1"}}

Note that the results here is a set of the datatype Part with the additional joined fields held as additional fields on Part.

So this seems to work perfectly, it is also transitive, for example:

(project 
    (join 
      (select #(= (:city %) "Paris") 
      suppliers) shipments {:number :supplier}) 
    [:name])

This will produce the name of the suppliers that have shipped parts and are located in Paris.

So all is good with relational algebra with datatypes in Clojure! Not quite, C J Date is a bit more picky than that, he really doesn’t like null values, his ideal system has no place for null values. Clojure has a null value, nil, and any value can be nil. So I can create a record in which all values are nil, for example:

clojure.core=> (Supplier. nil nil nil nil) 
#:Supplier{:number nil, :name nil, :status nil, :city nil}

Oh no - there are unknown values! We could create a factory function that ensured that you can’t pass in any nils but you can still do this:

clojure.core=> (assoc (Supplier. 1 "A" 20 "London") :number nil)
#:Supplier{:number nil, :name "A", :status 20, :city "London"}
clojure.core=> 

So nil can still worm its way in to the datatype and we’ve introduced unknown values into our relational algebra which we probably don’t want to do. So more work is needed to satisfy C J Date. However, the core operations from clojure.set have done remarkably well and it could easily sit in the core of a relational system with some wrapping to keep the evil nil at bay.

Having had this initial reaction, I still decided to dig through the documentation a bit, and spotted an interesting nugget - they’ll let you do geospatial searches. I’d been tinkering around with the idea of playing with this, especially for use with my shiny new Android phone, and I had an a idea for a little app to show you “nearby art” i.e. search with the V&A’s API for the nearest bit of art.

I did this mostly in Javascript, doing XMLHttpRequest’s for JSON chunks of the API. There’s also a block of Python code that needs to run on a server, but that’s entirely to get around the issues of XMLHttpRequest only allowing same-server requests. It first uses navigator.geoLocation (official spec, easier documentation) to get the user’s location, then does two V&A queries - the first to get a list of local objects, and the second to get more info on the first object.

One thing you have to be careful about is that this can break in various ways. The most obvious is a lack of navigator.geoLocation (any version of IE, and all not-latest versions of most other browsers), and another is if the user denies access to their location data. This does make navigator.geoLocation unsuitable for general use currently, but it’s a useful source of data when there is support.

The full app is over here and the source is here.

To start with here is a slight re-write of the structure I used last time to record the coverage:

(def coverage-records (ref nil))

(defn add-record [fn-name record]
  (dosync (alter coverage-records assoc (str fn-name) record)))

(defn get-record [fn-name]
  (get @coverage-records (str fn-name)))

(defn covering [form fn-name]
  (dosync (alter (get-record fn-name) assoc form 0)))

(defn- inc-map [map key]
  (if (contains? map key)
    (assoc (dissoc map key) key (inc (get map key)))
    map))

(defn inc-coverage [form fn-name]
  (dosync (alter (get-record fn-name) inc-map form)))

This code uses a single map to store a coverage record, another map, for each function that is being measured. Functions are provided for adding records and incrementing the coverage.

I am using the same set of functions to wrap and instrument the s-expressions that define the function, with a few minor modifications. I am indexing the s-expressions with a number so as to distinguish between any identical s-expressions within the function. This set of functions is still incomplete since there are lots of Clojure code that will get wrapped incorrectly, I will be rectifying this sometime in the near future.

(declare wrap-seq)
(defn wrap
  [form idx fn-name]
  (cond
    (seq? form) (
      let [key (str idx ":" form)]
        (covering key fn-name)
        (list ‘do `(inc-coverage ~key ~(str fn-name)) (wrap-seq form idx fn-name)))
    :else form))

(defn- indexed [coll] (map vector (iterate inc 0) coll))

(defn wrap-seq [coll count fn-name]
  (for [[idx elt] (indexed coll)] (wrap elt (+ count idx) fn-name)))

The previous article used a macro to generate Clojure code that was instrumented, this time I will load the source code for the function, wrap it and then evaluate it. This is done by the get-source function in clojure.contrib.repl-utils combined with a call to format and load-string, My code that interprets the source code is fragile and needs more work, it won’t handle functions with multiple bodies for example, but it demonstrates the principle. I am using a structure to hold the instrumented function and the coverage record for this function.

(defstruct wrapper :wrapped-fn :coverage-record)

(defn wrap-fn
  [f cv-rec]
  (let [s (read-string (get-source f))
        fn-name (first (drop 1 s))
        args (first (drop 2 s))
        body (last s)]
    (add-record fn-name cv-rec)
    (load-string (format “(fn %s %s)” args (wrap body 0 fn-name)))))

(defn wrap-function [f]
  (let [coverage-record (ref nil)]
    (struct wrapper (wrap-fn f coverage-record) coverage-record)))

So now we can try the code out at the REPL:

user=> (def x (wrap-function 'test1))   
#'user/x
user=> x
{:wrapped-fn #, :coverage-record #}

Here I have wrapped the function test1 and we can see the coverage structure returned, consisting of the wrapped function and the coverage record, keyed by the single s-expression with a count of 0. If I now use the wrapped function like this and examine the coverage structure the count should increment to 1.

user=> ((:wrapped-fn x) 1 2)            
-1
user=> x
{:wrapped-fn #, :coverage-record #}

So we now have a partially functional coverage tool and no macros have needed to be written. To complete this tool I just need to tidy up the wrapping and source code reading functions and provide some sort of binding macro so that we can call the fn with

(test1 1 2)

((:wrapped-fn x) 1 2)

I’d run into a few problems with it however, primary being that xwd doesn’t play nice with compositing, and so my Docky instance at the top of the screen looks like a big black patch across the screenshots, which occasionally blocks useful information.

I’ve therefore written a replacement, called LittleBrother, using primarily Python and gtk. Initially I was just following the instructions on the PyGtk FAQ on how to capture your screen, but then I figured I could do a bit more. So, we now have a little status bar icon that lets you open up the screenshots folder (using xdg-open so the right file manager gets used), and instead of having to hardcode in the times when you’re in the office, it instead uses the DBus interface of the Gnome screensaver (patches for other screensavers welcomed) to detect when the screensaver is active, and only take captures when it’s inactive. I found the D-feet DBus introspection tool particularly useful for playing around with this step.

Code is available on Github, and there’s even a Debian package available as well.

This document defines a date and time format for use in Internet protocols that is a profile of the ISO 8601 standard for representation of dates and times using the Gregorian calendar.

We needed to be able to robustly transfer timestamps between languages (Javascript and Python, chiefly) without getting tangled up in timezone troubles or complex ambiguous parsing problems.

Our code provides

• simple, standard, robust, cross-language (e.g. Javascript) format for parsing and printing time stamps
• a standard no-frills “UTC” tzinfo class and singleton instance
• a standard no-frills fixed-offset tzinfo class
• other utilities for helping write robust timezone-aware time manipulation code

Examples

These examples are taken from the doctests/docstrings in the module source itself. See the module documentation for many more informative examples.

Parsing a timestamp, with timezone support and timestamp equivalence:

>>> midnightUTC = parse_datetime("2008-08-24T00:00:00Z")
>>> oneamBST = parse_datetime("2008-08-24T01:00:00+01:00")
>>> midnightUTC == oneamBST
True

Printing a timestamp:

>>> oneamBST.isoformat()
'2008-08-24T01:00:00+01:00'
>>> parse_datetime("2008-08-24T00:00:00.123Z").isoformat()
'2008-08-24T00:00:00.123000+00:00'

Downloading the code

The code is available on github. It’s MIT-licensed.

• Browse the code here
• Download a tarball of the latest code

You can also install the module directly from github using pip:

pip install -e git://github.com/tonyg/python-rfc3339.git#egg=rfc3339

I have been experimenting with Clojure and it has a nice testing library but the only way I have found to measure coverage is to use Cobertura and squint at the generated class names to work out what I haven’t covered. So I have started on the road to creating a coverage tool for Clojure written in Clojure. This blog entry contains my initial steps and design sketches towards creating the tool.

Ideally I’d like to measure my coverage like this:

(ns some.tests
  (:use [some.functions])
  (:use [coverage.cover])
    (:use [clojure.test]))

(deftest test-something
    (cover ‘(foo bar)                   ; cover foo and bar
      (is (= [1 2 3] (foo 1 2 3)))
        (is (= [2 4 6] (bar 1 2 3)))

        (is (= [100 100] (coverage foo))) ; check coverage
        (is (= [100 100] (coverage bar)))

        (store-coverage)))                ; store report to file

That isn’t functional code yet and may not be what I finally produce, it might not even be valid Clojure since I haven’t even bashed it into a REPL yet.

As Clojure is a Lisp, and we are testing our own code - i.e. we have the source code - we should be able to measure the coverage using Clojure. This can achieved by rebinding the functions under test with instrumented functions; this is what I envisage the cover function is doing:

(cover '(foo)
  ; all calls to foo here actually call foo'
)

Which can be achieved using a binding function:

(binding [a b)]
    ; all calls to a here are actually calls to b
)

So assuming I can write the cover function I will need to instrument a function. Let’s start simply and ignore the complicated stuff such as conditional statements and multiple bodies until later. If we have this simple function

(defn foo [a b c] (+ a (- b c)))

I would like an instrumented version to wrap each s-expression with a function that records that the s-expression has been called, for now we will just print a message.

(defn foo-wrapped [a b c]
  (do
      (prn :wrap)
        (+ a
          (do
            (prn :wrap)
              (- b c)))))

Testing that in the REPL seems to reveal that I am moving in the right direction:

user=> (foo 1 2 3)
0
user=> (foo-wrapped 1 2 3)
:wrap
:wrap
0
user=>

To convert the first function to the instrumented function I need a couple of auxiliary functions and a macro:

(defn record [] (prn :wrap))

(declare wrap-seq)
(defn wrap [form]
  (cond
      (seq? form)
          (list ‘do (record) (wrap-seq form))
        :else form))

(defn wrap-seq [xs]
  (for [x xs] (wrap x)))

(defmacro cover
  [args & body]
    (let [fn-name (when (symbol? args) args)
          args (if fn-name (first body) args)
                body (if fn-name (next body) body)]
      (fn ~@(if fn-name (list fn-name args) (list args))
          ~@(wrap (first body)))))

If you know some Clojure that code demonstrates some of the problems I haven’t solved yet when measuring coverage, for example the :else clause will need to be covered as well as functions in binding statements. Testing that code in the REPL gives us a reasonable result:

user=> (def x (cover [a b c d] (+ a (- b c (* a d)))))
#’user/x
user=> (x 1 2 3 4)
:wrap
:wrap
:wrap
-4
user=>

So if I now replace the record function with something else I may be able to generate some coverage for a function. I am going to use a map held by a ref to store the coverage information, a function to register a particular s-expression with the ref and a function to increment the count.

(def coverage-records (ref nil))

(defn covering [form]
  (dosync (alter coverage-records assoc form 0)))

(defn- inc-map [map key]
  (if (contains? map key)
    (assoc (dissoc map key) key (inc (get map key)))
    map))

(defn inc-coverage [key]
  (dosync (alter coverage-records inc-map key)))

Lets test that:

user=> @coverage-records
nil
user=> (covering :x)
{:x 0}
user=> (covering :y)
{:y 0, :x 0}
user=> (inc-coverage :x)
{:x 1, :y 0}
user=> (inc-coverage :y)
{:y 1, :x 1}
user=> (inc-coverage :x)
{:x 2, :y 1}
user=> @coverage-records
{:x 2, :y 1}
user=>

If I modify our wrapping functions I can create the coverage records ready to be incremented.

(defn wrap [form]
  (cond
    (seq? form) (do
      (covering (str form))
      (list ‘do `(record) (wrap-seq form)))
    :else form))

Testing that code:

user=> (def x (cover [a b c d] (+ a (- b c (* a d)))))
#’user/x
user=> @coverage-records
{”(* a d)” 0, “(- b c (* a d))” 0, “(+ a (- b c (* a d)))” 0}
user=>

And that is as far as I have got. I need to change my record function to increment the coverage records, make my wrap function cope with conditionals and other complexities and write my rebinding function. Hopefully next month I will have moved it along a little bit further.

Read the full story

Enter dh-builddep-metapackage (the naming is inline with the names used by the debhelper scripts used for other debian packaging stuff). dh-builddep-metapackage builds build-dependency metapackages in order to ease package management for package rebuilders. In effect, it builds a “<package name>-builddep” package that has no content, but depends on everything that the existing package build-depends on. By using dh-builddep-metapackage to create metapackages rather than using “apt-get build-dep”, I keep a record in the package management about why I need a particular development package, and can remove the dependant packages when I’m no longer working with the relevant source package.

I think someone’s done this before, but some work with Google didn’t find it, and it was an interesting exercise anyway. Standard usage is “dh-builddep-metapackage -b <package name>”, which will create the metapackage data and build the package for you using dpkg-buildpackage. A folder called “<package name>-<package version>” will be created in the current local directory, and if an existing folder exists then dh-builddep-metapackage will refuse to overwrite it (unless you give the -o/–overwrite option).

So far I’ve used it a few times, and it’s quite nice to have those packages around to remind me about the dependencies. I’m considering building a full repository with *-builddep packages for everything in the Debian archives, but I’d like to make sure I’d use this enough first.