After spending a bit over three months at LShift, I am proud to leave LShift’s mark in the Minecraft Universe.

Frolicking over Minecraft’s cubic pastures and passing by interesting arrangements of hovering dirt blocks suspended in mid-air is all in a Minecrafter’s day’s work. But if you ever see light-blue wool blocks hanging around in the air, you can be sure that someone’s been . . . Shifty . . .

The ones you see in the picture above, in fact, have been put into the Minecraft world by a tool I wrote in Haskell. In this multi-part series, I want to share with you how I did it.

An Executive Summary

The theme:

What does it take to carry an idea from conception to realisation in Haskell?

And here are the requirements I worked with. The task is to write a Haskell program that takes

• a path to a PNG image
• path to a Minecraft saved world

and then it

• quantises the pixel data fron the PNG image into Minecraft coloured wool blocks (to a palette of a dazzling 16 colours!!)
• inserts these coloured wool blocks into the world at a height 5 blocks above the player’s head

Simple? Not quite. As with any software project — even one as small as this — the devil is in the details. The Minecraft saved game format turns out to have an interesting structure involving offsets, some compression and even some odd (even some odd? :/)  ’nybble endianness’ in places.

My Motivation

In case you are wondering, there are already many really good libraries for editing and manipulating Minecraft saved worlds (and no, I’m not talking about the diamond axe), so the concept of my tool is nothing new.

To use those tools directly for my aims presents little challenge. Being a bit of a Haskell enthusiast, this project presented itself as a fun way to try Haskell out for size on problems resembling those you see in the real-world, such as dealing with and manipulating binary data.

The Road Ahead

Throughout the series, I aim to uncover details as we require them, just as I did while I explored the problem space. At a glance:

• Haskell stubbing for fun and profit: We (pretend!) to practise test-driven development by making the success of reading and writing Minecraft saved world files (called Region files) a test property
• Dealing with compression using Functors and Phantom Types: We design the main data types for maximum programmer comfort, paving the road for operations on arrays of block data.
• Serious Binary Serialisation: We implement code to read and write Region files using the Get and Put monads from Data.Binary, and ensure a roundtrip succeeds
• Update your Chunks everywhere using SYB and SYZ: Time to change the world! We write code to perform somewhat troublesome chunk updates. We also extract the player’s current coordinates in the world from Level data
• Fun turning Pixel Data into Wool using Codec.DevIL: We read in image data using Codec.DevIL, write a simple quantisation function, and build the final executable

I must admit I didn’t actually practise TDD, but the way I went about it actually more in line with the strong-typing mantra:

If it compiles, your code’s probably doing something useful.

Though whenever I found that my code didn’t work immediately after making it compile, the fall-back is simply to write some tests to help debug the problem.

And that, dear reader, is the principal matter of the next post. I hope you’ll enjoy it!

There are a number of type inference techniques for dynamic languages - k-CFA, demand-driven type inferencing with subgoal pruning, RoelTyper. I’m going to use a very simple technique.

First, some terminology. In Smalltalk, “protocol” usually means one of two things: either “what messages does this object understand?” or “does this object understand the Foo protocol?”, where Foo might be “Stream”, or “Collection”. We’re going to use the latter meaning. In particular, given a JUnit-like theory, we want an answer to the question “What objects - what instances of what classes - satisfy the protocol sent to the datum injected into this theory?”

With a decompiler to hand, it’s easy enough to generate an AST of the theory over which we can walk: walk the MessageNodes and look for things with a receiver 't1' which will be the name given to the first temporary variable, i.e., the argument of the unary method. This won’t work for anything hidden through a #perform: - think of this as the eval of Smalltalk code.

    ParseNodeVisitor subclass: #SenderToArgCollector
    instanceVariableNames: 'selectors classSelectors'
    classVariableNames: ''
    poolDictionaries: ''
    category: 'SqueakCheck-SUnit'

    visitMethodNode: aMethodNode
        classSelectors := Set new.
        selectors := Set new.
        ^ super visitMethodNode: aMethodNode.

    visitMessageNode: aMessageNode
        (aMessageNode receiver name = 't1')
            ifTrue: [selectors add: aMessageNode selector key].
        (aMessageNode receiver isMessageNode
            and: [aMessageNode receiver selector key = #class]
            and: [aMessageNode receiver receiver name = 't1'])
            ifTrue: [classSelectors add: aMessageNode selector key].
        ^ super visitMessageNode: aMessageNode.

    selectors
        ^ selectors.

    classSelectors
        ^ classSelectors.

and invoked by our TheoryTyper (which could be called a DuckFinder…)

    messagesSentToDatum: aUnaryCompiledMethod
    "Answer a pair of Sets of all the message selectors sent by this method to its argument.
    The first Set contains messages sent to the argument, and the second contains messages
    sent to the argument's class."
    | collector |
    collector := SenderToArgCollector new
        visitMethodNode: (Decompiler new
            decompile: aUnaryCompiledMethod selector
            in: aUnaryCompiledMethod methodClass).

    ^ {collector selectors. collector classSelectors}.

Let’s try out our new toy. To recap, we wish to write a theory and have the system automatically find the right types of things to test the theory. So let’s try the “monadic laws”:

    Object subclass: #TheoryTyper
    instanceVariableNames: ''
    classVariableNames: ''
    poolDictionaries: ''
    category: 'SqueakCheck-SUnit'

    monadsObeyLeftIdentity: m

    self
        assert: (m class return: m value) >>= [:t | m class return: t]
        equals: ([:t | m class return: t] value: m value)

    monadsObeyRightIdentity: m

        self assert: m equals: (m >>= [:a | m class return: a])

And lo! our duck-finder says, with the Maybe and Either monads loaded:

    TheoryTyper new typeOfDatum: (MonadTheories >> #monadsObeyLeftIdentity:)
    "=> a Set(Maybe Either)"

The keen-eyed will notice a law missing from the above - the associativity law for monads. Given monadic blocks f and g - that is, unary blocks that take some value and return a value wrapped up in whatever monad you’re using - we can express this law as

    monadsObeyAssociativity: m
    | f g |

    self
        assert: ((m >>= f) >>= g)
        equals: (m >>= [:x | (f value: x) >>= g])

but from where do we get f and g? Our naive duck-finding fails: we would need to extend our “type inference”. One possible (and fairly ugly) solution is to make m’s class responsible through helpers: add sampleBlockF and sampleBlockG messages to the protocol and we could write:

    monadsObeyAssociativity: m
    | f g |

    f := m class sampleBlockF.
    g := m class sampleBlockG.
    self
        assert: ((m >>= f) >>= g)
        equals: (m >>= [:x | (f value: x) >>= g])

Also, typing m is a bit harder than in the previous examples:

    TheoryTyper new typeOfDatum:
        (MonadTheories >> #monadsObeyAssociativity:)
    "=> a Set(Either Nothing Maybe Left Right Just)"

because all we have to work with is the send of #>>=. Arguably it would be clearer to return a Set(Either Maybe) because the other classes are subclasses of these two.

However, despite the limitations of this technique, one can express theories in a nicely modular way. The monadic laws will simply run against any monadic classes (that is, any classes that understand #>>= on the instance side and return: on the class side) present in the image.

Reflect on that statement. Apparently Tim has a strong dislike for a concept found in lots of programming languages (even brainiac languages like Haskell) and successfully used in millions of programs. He must be crazy I wouldn’t like to have a discussion with him about something contentious like tabs versus spaces.

I understand that the world is an uncertain place and that programs need to represent uncertainty but I don’t think null can be the correct way.

For example, consider this question: how many pets do you own? Is null a suitable answer?

Taking Java as the target for this polemic, which of Integer or int is the best way to represent the number of pets you have?

Hint, one of those types can’t be null. Bonus question: what happens if you unbox an Integer that holds a null value? Research topic: why does Java need boxed numeric types?

Some people have told me that they would use a type that can be null because they don’t know how many pets that I own. Sorry, but I don’t think that is a good answer!

How would your program be different if you let zero be the default value for a quantity instead of null?

If their is uncertainty in a property would it be polite to let me know?

In order to guarantee that your program works correctly do you need to check that every value is non-null before proceeding?

Is your code very hard to read and comprehend because of the huge number of null checks?

Or do you ignore that because all of your unit tests pass?

You do pass nulls into all your methods via your unit tests don’t you?

Moving on!

Now if we establish that Tim has 23 pets, how do we associate them with Tim?

I’d probably use List<Pet>.

Can a Collection contain nulls?

Can a collection be null?

Should you use null to represent an empty collection?

How would your program be different if you used Collections.emptyList() or similar instead of null?

Should we stop programming and hope that JSR-305 makes it into the next version of the JDK?

Is their a refactoring in your IDE that will make Tim and his nulls go away?

Is it easier to press generate getters and setters in your IDE because no one you know would pass in a null anyway at least not whilst you are still responsible for the project?

Do you think Tim is a wee bit upset about nulls in Java code?

Should he go and code in a more modern language?

Not necessarily! Guava can be quite helpful.

You don’t need to use Scala to get an Option type, Guava has Optional

Guava has immutable collections that reject nulls.

Guava has handy pre-conditions that you can use to prevent nulls entering at your constructors and setters.

Have I ranted for long enough about null?

Yes.

Luckily, Rhythmbox has a nice plugin system, so I can fix that. Having done this sort of thing a little bit before, but not for a while, I started at the plugin writing guide. As it turns out, in the 2 weeks between starting writing this and doing this post, they’ve updated that guide a lot, but at the time it was very out of date. My major guide was in fact the ‘Remember the Rhythm’ plugin off of the 3rd party plugins list. The top of the plugin may look a little unfamiliar to anyone whose done any Python Gtk programming before, and have many of you wondering what this “gi.repository” thing is. “gi” in this case stands for GObject Introspection, a new and shiny way to interface between object-orientated C written with GObject (i.e. a lot of things in the Gnome project) and your favourite scripting language. More specifically, instead of writing language-specific bindings (although there can be language-specific extensions), a library writer can now just write for the GObject introspection system and all of the fans of language Foo can just write bindings for the introspection stuff, and everyone gets to do less work. This isn’t quite introspection in the same way as most people know it, as the tool starts from scraping C header files (plus some custom overrides), but it’s still a nifty little item.

(Side note: GObject Introspection’s main claim to fame is the Gnome Shell, the major feature of Gnome 3, which uses GObject Introspection with the Javascript bindings heavily to do a lot of its work)

The other new item there would be the Peas system, which is a general-purpose plugin system for GObject-based systems, leveraging the GObject Introspection stuff to let you write multi-language plugins (although Rhythmbox always had Python support).

Other useful items along the way were the Python Gtk+ 3 tutorial for the basics, and the main Gtk+ manual (written for the C interfaces, but you can kinda figure things out). You’re also going to need the Rhythmbox developer manual and for when you want to do funky things like a play/pause icon, you’ll need the Python Cairo bindings (or the relevant PyGtk FAQ entry when you get stuck).

As always, there’s Debian packaging, and I’ve even added a downloadable prebuilt .deb from the Github repo

Clojure’s abstract bindings provide much the same capability - (let [[h & t] ‘(1 2 3)] <do stuff>) - modulo the difference between pattern matching and unification, of course.

There’s a subtlety in something like [H|T] = [1,2,3], at least, if your lists aren’t built of nested cons cells. Consider Smalltalk arrays. Suppose we have some ListUnifier that will rip a SequenceableCollection’s head off, like #(1 2 3). We’d like the tail to unify with #(2 3) in other words. But that’s not a node in the original structure - it’s an entirely artificial node we wish to construct from the original collection. Firstly, how can we unify with only part of a structure and, secondly, how can we determine a solution from that partition?

Let’s try model the parts:

    DestructuringUnifier subclass: #ListUnifier
        instanceVariableNames: 'head tail'
        classVariableNames: ''
        poolDictionaries: ''
        category: 'Unification-Destructuring'.

    ListUnifier >> head: anObject tail: anotherObject
        head := anObject.
        tail := anotherObject.

    ListUnifier class >> headNamed: headSymbol tailNamed: tailSymbol
        ^ self new head: headSymbol asVariable tail: tailSymbol asVariable.

    "Various helper constructors like #head:tailNamed:, #head:tail:, etc.
     elided for brevity."

Then we can write the original Prolog statement as

    (ListUnifier headNamed: #x tailNamed: #y) =? #(1 2 3)

As mentioned above, first we want to be able to construct an equivalence relation on the above (or, expressed differently, partition the set of nodes in the structure together with the artificial nodes we create) such that #x asVariable and 1 are in the same class, and ditto for #y asVariable and #(2 3).

    unificationClosureWith: anObject in: termRelation
        | h t partition |
        anObject isMetaVariable
            ifTrue: [^ termRelation union: self with: anObject].
        anObject isCollection
            ifFalse: [^ self failToUnifyWith: anObject].
        anObject isEmpty
            ifTrue: [^ self failToUnifyWith: anObject].

        h := head isCollection
            ifTrue: [anObject first: head size]
            ifFalse: [1].
        t := head isCollection
            ifTrue: [anObject allButFirst: head size]
            ifFalse: [anObject allButFirst].
        partition := head unificationClosureWith: h in: termRelation.
    ^ tail unificationClosureWith: t in: partition.

The mild complication around head isCollection lets us support a head that is itself a collection. So let’s check that we can construct a partition using parts of things:

    | left right partition |
    left := (ListUnifier headNamed: #x tailNamed: #y).
    right := #(1 2 3).
    partition := VariableTrackingUnionFind
        usingArrayType: PersistentCollection
        partitioning: Dictionary new.
    partition := (partition find: left)
        unificationClosureWith: (partition find: right) in: partition.
    partition elementsOfClass: #x asVariable. "=> {1 . (#Variable #x)}"
    partition elementsOfClass: #y asVariable. "=> {#(2 3) . (#Variable #y)}"

We see that the partition that originally would only hold nodes in the structures may now hold parts of the original structure.

The original algorithm for determining the most general unifier from some partition as described in Baader & Snyder (pp. 461-462) runs the solution finder starting from the left operand in the unification. Consider the partition we have above. What elements are in the equivalence class of ListUnifier? Well, just the ListUnifier itself! Clearly we need to adjust the solution finder a bit. The obvious approach would be to start the solution-finding from an element in each class, and merge the partial solutions:

    findSolutionFor: aVariableAvoidingUnionFind
    ^ aVariableAvoidingUnionFind
        inject: MostGeneralUnifier new
        into: [:mgu :node |
            mgu addAll: (self new
                findSolutionFor: aVariableAvoidingUnionFind
                starting: node)]

where #addAll: merges the various MostGeneralUnifiers generated and #inject:into: folds over the representative node in each equivalence class. (Remember, a union-find always has a representative for each equivalence class, namely, myPartition find: someObject.) And it works, at the cost of turning a linear algorithm into a (worst case) quadratic one:

    | left right |
    left := (ListUnifier headNamed: #x tailNamed: #y).
    right := #(1 2 3).

    left =? right "=> MostGeneralUnifier((#Variable #x)->1 (#Variable #y)->#(2 3) )"

But “finding a solution” really means “to what must we assign each variable?”. So we can at least speed things up by only solution-finding in those classes in which variables occur:

    findSolutionFor2: aVariableAvoidingUnionFind
    ^ aVariableAvoidingUnionFind variableContainingClasses
        inject: MostGeneralUnifier new
        into: [:mgu :node |
            mgu addAll: (self new
                findSolutionFor: aVariableAvoidingUnionFind
                starting: node)]

This makes finding a solution O(NM), where N is the number of nodes in the structure, M the number of classes containing variables.

    | matcher mgu |
    matcher := OrUnifier
        left: (TreeNode left: #x asVariable)
        right: (TreeNode right: #x asVariable).

    mgu := matcher =? (TreeNode left: (Leaf value: 1)).
    mgu at: (#x asVariable) "=> (Leaf value: 1)".

    mgu := matcher =? (TreeNode right: (Leaf value: 1)).
    mgu at: (#x asVariable) "=> (Leaf value: 1)".

Easy enough… until one tries to use an OrUnifier as an operand on the right hand side. See, as the unification progresses, if the first option fails, you’d like to backtrack part of the equivalence relation calculation, and with the imperative union-find in Nutcracker that’s not possible. What to do, what to do?

The standard solution is to reach into one’s toolbox of functional data structures. Sadly, noone knows (as far as I can see, at least) how to implement a functional union-find. At least, not an efficient one. However, Conchon and Filliâtre tell us how to implement a persistent union-find.

The implementation - originally in ML - uses a persistent array to get its rollbackability. Further, it uses “rerooting”, a trick Henry Baker wrote up, to improve efficiency. I couldn’t improve on the pictures Conchon and Filliâtre use to illustrate rerooting, so I won’t try, and just point you to their artwork, pp. 18-27. The structure makes massive use of side effects, but presents an apparently purely function API. First, the basic data structure:

    type 'a t = 'a data ref
    and 'a data =
        | Arr of 'a array
        | Diff of int * 'a * 'a t

Note the ref there - it’s an updatable reference to something. Since everything’s mutable by default in Smaltalk, I tried ignoring the ref and just translate things. However, I quickly ran into difficulties. The “massive use of (hidden) side effects” quickly bit me (see section 4 of the paper), as I attempted to translate the following:

    let set t i v = match !t with
        | Arr a as n ->
            let old = a.(i) in
            a.(i) <- v
            let res = ref n in
            t := Diff (i, old, res);
            res
        | Diff _ ->
            ref (Diff (i, v, t))

It all looks quite simple. But look at t := Diff (i, old, res). Ignoring it at first, the obvious translation would be (ignoring noise like class declarations):

    Diff >> set: index to: anObject
        ^ Diff index: index value: v in: self

    Arr >> set: index to: anObject
        | old res ref |
        old := a at: i.
        a at: i put: v.
        res := self.
        self become: (Diff index: i value: v in: self)
        ^ self.

Did you feel a shiver there? #become: is deep Smalltalk magic. Its operation is simple enough: it swaps two object pointers. Here, we will change self to a new object. Oh, did I say “swap two object pointers”? I meant to say “swap two object pointers throughout the entire image”. Deep, dangerous magic indeed. And I thought, as I tried to figure out what was going on, that there had to be an easier way. What if I modelled the ref itself?

    Object subclass: #Ref
        instanceVariableNames: 'value'
        classVariableNames: ''
        poolDictionaries: ''
        category: 'PersistentUnionFind'.

    Ref class >> wrapping: anObject
        ^ self new wrapping: anObject.

    Ref >> value
        ^ value.

    Ref >> value: anObject
        value := anObject.

    Ref >> wrapping: anObject
        value := anObject.

Now you can have an immutable reference to something that, itself, may change to what it points. (Yes, that sounds a lot like “well done, you’ve invented a pointer!”) With that in hand, let’s hide the gory bits - Arr and Diff - behind a nice clean PersistentCollection interface:

    Ref subclass: #PersistentCollection
        instanceVariableNames: ''
        classVariableNames: ''
        poolDictionaries: ''
        category: 'PersistentUnionFind'.

    PersistentCollection >> at: index put: anObject
        t := value. "The equivalent of !t"
        ^ t isDiff
            ifTrue: [Diff index: i value: anObject in: self]
            ifFalse: [ | old |
            old := t array at: index.
            t array at: index put: anObject.
                self value: (Diff index: i value: old in: self)]

which doesn’t look too bad, in comparison to the original!

The code’s published at SqueakSource, in the PersistentUnionFind package:

    Installer ss
      project: 'Nutcracker';
      install: 'PersistentUnionFind'.

The core problem I’m trying to address is as follows: you find a new artist, and you really like them and buy all their CDs. Problem is, is that there’s nothing notifying me when they do a new release, and given new releases are a relatively rare event, I forget to check… essentially I want Songkick, but for CDs (or MP3 albums, but the principle still applies).

Enter Missing Albums. It works as follows:

• Trawls through your existing music collection, and finds all the bands you’ve got at least 4 tracks of (which tends to eliminate most of the uninteresting bands), and gets an album list for each based on the tags of those files (presence of any track from an album is assumed to imply knowledge of that album)
• Grabs the Musicbrainz data for those bands, and establishes a list of albums for each.
• Find the dates for each existing album for a band, determine the newest one, and get the list of all albums for that band after that date.
• Grab Amazon price data and album cover for each newer album and spit out a list of albums in reverse chronological order. Genshi is used to give this nice formatting.

Results look like this:

The tool manages to solve my core problem i.e. finding out about new albums, but it’s still got a number of flaws:

• Musicbrainz is quite slow, and so this is a batch processed command-line app for the moment.
• The only identifier we’ve got for a band is it’s name, and that’s non-unique in a large enough set of cases. Brian Whitman’s excellent article on the issues with Facebook’s music ID system illustrates this well, and I’ve got some pretty nasty cases just in my collection (e.g. I have both tracks from the Belgian rock band and Trance artists called “Deus”, and there’s another 2 entries for it in Musicbrainz; there are 5 entries all called “Muse” and 7 called James). In this tool, I’ve hacked around the issue by ignoring anyone with no albums listed on Amazon, which tends to thin things down to just the big commercial bands.
• My artist tagging scheme mostly lists bands starting with the word “The” as “Foo, The” rather than “The Foo”, but telling that both are the same is again a bit of a problem due to the same disambiguation problem
• Sometimes a band isn’t present at all in Musicbrainz, or there’s no Amazon data for an album. I’ve solved this by either a) adding the data into Musicbrainz, or b) adding artists to an “ignore” list. The missing-albums.py script takes as one of it’s arguments an “overrides” file which is an ini-style file containing potentially two sections; first is “artist”, which rewrites any artist names I’ve drastically misspelt and the second is “ignore” which is a list of artists to skip (e.g. my meta-artist name of “Theme” which I’ve marked most of the TV/film tracks I’ve got)

On the other hand, all these issues aren’t too awful for a command-line tool. If I was trying to make a slick web interface for this, they’d be major problems, but I’m willing to skip the odd dodgy result as the results are very usable already, and have resulted in some purchasing already.

TODO

• Work with last.fm data instead of trawling local music files

My test class is simple, I call him Thing, he has two fields name and content, he isn’t a Java bean because he doesn’t need to be (maybe your classes don’t need to be Java beans either!), here he is:

public class Thing {
    public Thing(String name, String content)   {
        this.name = name;
        this.content = content;
    }

    final public String name, content;
}

We will use JUnitBenchmarks to test my theory that you can be simpler and faster than Jackson. JUnitBenchmarks allows unit tests to be run multiple times and measurements taken, it also allows the code to be warmed up so any JIT compilation should have been carried out before measurements are taken. I’ve set my tests to run a warmup period of 50 iterations followed by 1000 measurement iterations. The Jackson code being tested looks like this:

public class JacksonStreamingSerialiser implements Serialiser {

    public String toJson(Thing thing) {
        StringWriter out = new StringWriter();
        try {
            JsonGenerator generator = jsonFactory.createJsonGenerator(out);
            generator.writeStartObject();
            generator.writeStringField("name", thing.name);
            generator.writeStringField("content", thing.content);
            generator.writeEndObject();
            generator.close();
        }
        catch (IOException e) {
            e.printStackTrace();
        }

        return out.toString();
    }

    private final JsonFactory jsonFactory = new JsonFactory();
}

When tested we get a mean measurement for writing 250 objects using Jackson of 0.95 seconds

json-lib is usually a worse performer than Jackson. Here is the equivalent code using json-lib:

public class JsonLibSerialiser implements Serialiser {
    public String toJson(Thing thing) {
        JSONObject object = new JSONObject();
        object.put("name", thing.name);
        object.put("content", thing.content);

        return object.toString();
    }
}

When tested we get a mean measurement for writing 250 objects using json-lib of 10.74 seconds. Not so good!

Who needs library code? Maybe that new fangled String.format will be quicker and simpler. Here is the code:

public class StringFormatSerialiser implements Serialiser {
    public String toJson(Thing thing) {
        return String.format("{\"name\":\"%s\",\"content\":\"%s\"}", thing.name, thing.content);
    }
}

When tested we get a mean measurement for writing 250 objects using String.format of 5.26 seconds. Better than json-lib but Jackson isn’t looking worried!

I guess that format string must be expensive to parse so lets try a StringBuilder. Here is the code:

public class StringBuilderSerialiser implements Serialiser {
    public String toJson(Thing thing) {
        StringBuilder builder = new StringBuilder();
        builder.append("{\"name\":\"").append(thing.name).append("\",\"content\":\"").append(thing.content).append("\"}");

        return builder.toString();
    }
}

When tested we get a mean measurement for writing 250 objects using StringBuilder of 0.91 seconds. Finally a winner, faster than Jackson but you had better quote those strings properly!

People have always told me that StringBuilder is unsyncronised so should be faster than an old fashioned StringBuffer so lets check. Here is the code:

public class StringBufferSerialiser implements Serialiser {
    public String toJson(Thing thing) {
        StringBuffer buffer = new StringBuffer();
        buffer.append("{\"name\":\"").append(thing.name).append("\",\"content\":\"").append(thing.content).append("\"}");

        return buffer.toString();
    }
}

When tested we get a mean measurement for writing 250 objects using StringBuffer of 0.60 seconds. Hang on! That is the fastest yet! What is going on?

What is going on is that I have been playing fast and loose with statistics by only presenting the mean times in seconds for each benchmark. By extracting the raw data and applying the power of statistics (well I looked at the distributions and standard deviations) it turns out you cannot tell the difference between the StringBuilder and the StringBuffer, so all is well, and String[Builder|Buffer] are both winners! Jackson is also a winner since for more complex object trees it will allow you to write more maintainable code than using a StringBuilder combined with loops and conditional logic and is almost as fast as a StringBuilder (or StringBuffer).

So what have we learnt? Firstly, use Jackson if you have to serialise your objects into JSON. Secondly, JUnitBenchmarks is a very handy library. Thirdly, if you don’t present a standard deviation with your benchmark results then your results may not mean what you think they mean.

Today, we’re going to see what else we can do with zippers.

When we implement a Huet-style zipper (in other words, we use a recursive structure based on one-hole contexts instead of directly using partial continuations), we implement a bunch of primitive behaviours - left, right, replace, insert child, and so on. Let’s assume we have all that.

The first obvious thing to do is to define a traversal. We need two essential elements for a traversal - a means of knowing when we’re done, and a means of moving to the next element. Now there’s a teeny hitch. Given a zipper over some hierarchical structure, it seems most natural to describe our traversal in terms of recursion - after all, we’re working with a recursive structure. Ruby doesn’t require proper tail call elimination (even though, apparently, some implementations support it). We don’t want stack overflows from too-deep recursion, so we’ll reach into our toolbox and haul out a trampoline:

    def trampoline(initial_value, &unary_block)
      result = unary_block.call(initial_value)
      while result.kind_of?(Proc) do
        result = unary_block.call(result.call)
      end
      result
    end

A trampoline will take some block, and invoke it with the given initial value. If the block yields a Proc (what the Lispers would call a thunk - a parameterless Proc that delays the evaluation of some chunk of code), it will invoke the block with this Proc, and repeat that process until the block yields something that’s not a Proc. Finally, it will return that non-Proc value. With tool in hand, we can now write a recursive pre-order traversal, confident that we won’t blow our stack even on large structures. Note that this particular zipper uses “safe” navigation, wrapping up results in the Either monad.

    # Perform a pre-order traversal, that is "this node, then a pre-order
    # traversal of my children, left to right".
    def next
      if not has_next? then
        return @zipper
      end

      if @zipper.branch?(@zipper.value) then
        return @zipper.down
      end

      right_sibling = @zipper.safe_right
      if right_sibling.right? then
        return right_sibling.value
      end

      # Return a Zipper if there's a next, with a distinguishable context
      # for the last element of the traversal.
      # This algorithm returns a thunk when it wishes to recurse.
      # The trampoline converts this CPS-like algorithm into one
      # that runs in constant space.
      trampoline(@zipper) { |z|
        parent = z.safe_up
        parent.either(->parent_z{
                        uncle = parent_z.safe_right
                        uncle.either(->z{ next z},
                                     ->unused_error{ next ->{parent_z}}) # Recur
                      },
                      ->unused_error{
                        # We've popped up the structure all the way to the root node.
                        z.new_zipper(z.value, EndOfTraversalContext.new(z.context))
                      })
      }
    end

It’s easy to wrap up next inside some object, and simply invoke the entire traversal with each:

    it "should process all nodes in a pre-order fashion" do
      tree = Tree.new(1, [Tree.new(2, [Tree.new(3, [])]),
                          Tree.new(4, [Tree.new(5, []),
                                       Tree.new(6, [])])])
      t = PreOrderTraversal.new(tree.zipper)
      answers = []
      t.each { |value|
        answers << value.value
      }
      answers.should == [1, 2, 3, 4, 5, 6]
    end

where each is defined thusly:

    def each(&block)
      map { |node|
        block.call(node)
        node
      }
    end

    # Return a same-shaped structure with the relevant mapping performed on
    # each node.
    def map(&unary_block)
      # It's ridiculous to store the previous zipper to avoid a one-past-the-end
      # error. It works, and it's simple, but it's _ugly_.
      prev = @zipper
      while has_next? do
        @zipper = @zipper.replace(unary_block.call(@zipper.value))
        prev = @zipper
        @zipper = self.next
      end
      prev
    end

We could just mix in Enumerable and get map and a whole lot more For Free… but Enumerable#map returns an Array, and I want map to return something of whatever type it was fed.

Of course there’s nothing stopping us from implementing other traversals - post-order, in-order (for binary trees), or even breadth-first. What’s nice is that none of these traversal strategies have any knowledge about the structure over which they move! They just know about the zipper… and the zipper knows nothing about the structure, except

• Can this node have children?
• What are this node’s children?
• Given a parent node and children nodes, how do I make a new node?

With a traversal in hand, we’re free to jump into the land of higher order functions - map, fold, and the like. Now of course, as we just saw, these traversals know nothing about the underlying structure. That means that while we can implement a fold -

    # Collapse some structure into some kind of value using an initial value,
    # and a binary block taking the thus-far-computed value (accumulator) and
    # the current node.
    def fold(initial_value, &binary_block)
      accumulator = initial_value
      each { |node|
        accumulator = binary_block.call(accumulator, @zipper.value)
      }
      accumulator
    end

- we will obviously have to tell the fold what to do. Actually, this is no different from your usual fold in Common Lisp or Smalltalk or Haskell. So given some kind of tagged tree with Node and Leaf elements, we could have

    it "should permit the folding of a structure according to a given block" do
      t = Node.new(:root, [Node.new(:left_subchild, [Leaf.new(1)]), Leaf.new(2)])
      PreOrderTraversal.new(t.zipper).fold(0) { |sum, node|
        sum + case node
                when Node then 0
                when Leaf then node.value
              end
      }.should == 3
    end

So here we see how to apply all manner of transformations - mapping a tree to another tree of the same shape, folding a tree, insert our fondest desire - with the core mechanisms remaining firmly decoupled from our types.

What to do? Random testing to the rescue! I cast around for Ruby ports of QuickCheck, and found two: rushcheck and rantly. Rushcheck hasn’t been wrapped up in a gem, so I decided to take rantly for a spin.

The basic structure of a rantly test is:

    property_of {
      my_special_generator
    }.check { |generator, values|
      my_special_property(generator, values)
    }

In other words, property_of generates some data, and the results are fed into check, where you describe the property you’re checking.

So, for example, a full spec might be

    class Rantly
      def peano(limit = nil)
        limit = 0..Peano::MAX_INT if limit.nil?
        Peano.from_i(integer(limit))
      end
    end

    module Peano
      describe "Peano" do
        it "should 0 < n" do
          property_of {
            peano(1..100)
          }.check { |n|
            Peano.zero.should < n
          }
        end
      end
    end

This example comes from a basic Peano number library I wrote for the purposes of playing with rantly. (I’m really not kidding about “basic”: I’ve intentionally limited the Peano numbers to the range [0, 1000] because I’m lazy.

Rantly supports Test::Unit, so you can always just subclass Test::Unit::TestCase and write your test as per normal with assert_equals and friends. Since I like RSpec I had to add a little helper:

    class RSpec::Core::ExampleGroup
      def property_of(&block)
        Rantly::Property.new(block)
      end
    end

rantly supplies a number of basic generators - integers, ranged integers, strings, booleans, chars, etc. - as well as various combinators, and scoped settings. Need an array of between 3 and 5 Peano numbers? No problem:

    # When you want to share the size between multiple generators ...
    sized(integer(3..5)) {
      array { peano }
    }

    # ... or when you don't need to.
    array(integer(3..5)) { peano }

rantly doesn’t use a polymorphic method for data generation, unlike QuickCheck’s use of typeclasses. It’s hardly difficult, of course, to roll your own such thing:

    class PNumber
      def self.generator
        Peano.from_i(Rantly.integer(0..Peano::MAX_INT))
      end
    end

Most importantly though, a test framework’s only as good as its output. If your assertions don’t result in decent error messages, you might as well not bother. So let’s say we have defined :== and :< for PNumbers. We would obviously also like :>. We write up a property (first!):

    it "should succ(n) > n" do
      property_of {
        PNumber.generator(0..3)
      }.check {|n|
        n.succ.should > n
      }
    end

When we run rake test we see:

    .
    failure: 0 tests, on:
    #
    F

    Failures:

      1) Peano should succ(n) > n
         Failure/Error: n.succ.should > n
         NoMethodError:
           undefined method `>’ for #>
         # ./test/peano_test.rb:96:in `block (3 levels) in ‘
         # ./test/peano_test.rb:93:in `block (2 levels) in ‘

We see a decent error message in the final output thanks to RSpec. Just as important - given that this is a test framework using random data - we see a counterexample. Subsequent runs, in this case, would give us different counterexamples. (”In this case” because, since we haven’t defined :> yet, every example is a counterexample!)

Let’s play around a bit, and half-implement :>:

    def > (peano)
      if peano.to_i.even? then
        not (self < peano) and not (self == peano)
      else
        false
      end
    end

Our output then looks like this:

    .
    failure: 0 tests, on:
    #>>>
    F

    Failures:

      1) Peano should succ(n) > n
         Failure/Error: n.succ.should > n
           expected: > #>>>
                got:   #>>>>
           Diff:
           @@ -1,2 +1,2 @@
           -#>>>
           +#>>>>
         # ./test/peano_test.rb:96:in `block (3 levels) in ‘
         # ./test/peano_test.rb:93:in `block (2 levels) in ‘

Or, “gosh darn, :> is broken for odd numbers!”

A final note: rantly has some dependencies, but fortunately not that many: rake (naturally), technicalpickles-jeweler, yaml.

In summary, rantly is a simple, easy to use random data generator that works nicely with Test::Unit and is easily extended to use RSpec. It comes with basic generators, and it’s easy to extend the generator support to arbitrary structures.