technology from back to front

Randomly testing Ruby

I recently ran into the need for testing the behaviour of a parser of a modelling language. The parser processes a number of model descriptions in gem files, as well as local definitions. Until recently, the parser would process the gems in an arbitrary order. However the language while ostensibly declarative, isn’t, because of a huge restructuring of the parser. As a result, some bugs lurk in the changed code. These bugs, because of the arbitrary processing order of the model gems, manifest on some machines and not others. Forcing an ordering on the gem processing masks the underlying issues, even while letting the users of the parser to get on with their lives.

What to do? Random testing to the rescue! I cast around for Ruby ports of QuickCheck, and found two: rushcheck and rantly. Rushcheck hasn’t been wrapped up in a gem, so I decided to take rantly for a spin.

The basic structure of a rantly test is:

    property_of {
      my_special_generator
    }.check { |generator, values|
      my_special_property(generator, values)
    }

In other words, property_of generates some data, and the results are fed into check, where you describe the property you’re checking.

So, for example, a full spec might be

    class Rantly
      def peano(limit = nil)
        limit = 0..Peano::MAX_INT if limit.nil?
        Peano.from_i(integer(limit))
      end
    end

    module Peano
      describe "Peano" do
        it "should 0 < n" do
          property_of {
            peano(1..100)
          }.check { |n|
            Peano.zero.should < n
          }
        end
      end
    end

This example comes from a basic Peano number library I wrote for the purposes of playing with rantly. (I’m really not kidding about “basic”: I’ve intentionally limited the Peano numbers to the range [0, 1000] because I’m lazy.

Rantly supports Test::Unit, so you can always just subclass Test::Unit::TestCase and write your test as per normal with assert_equals and friends. Since I like RSpec I had to add a little helper:

    class RSpec::Core::ExampleGroup
      def property_of(&block)
        Rantly::Property.new(block)
      end
    end

rantly supplies a number of basic generators – integers, ranged integers, strings, booleans, chars, etc. – as well as various combinators, and scoped settings. Need an array of between 3 and 5 Peano numbers? No problem:

    # When you want to share the size between multiple generators ...
    sized(integer(3..5)) {
      array { peano }
    }

    # ... or when you don't need to.
    array(integer(3..5)) { peano }

rantly doesn’t use a polymorphic method for data generation, unlike QuickCheck’s use of typeclasses. It’s hardly difficult, of course, to roll your own such thing:

    class PNumber
      def self.generator
        Peano.from_i(Rantly.integer(0..Peano::MAX_INT))
      end
    end

Most importantly though, a test framework’s only as good as its output. If your assertions don’t result in decent error messages, you might as well not bother. So let’s say we have defined :== and :< for PNumbers. We would obviously also like :>. We write up a property (first!):

    it "should succ(n) > n" do
      property_of {
        PNumber.generator(0..3)
      }.check {|n|
        n.succ.should > n
      }
    end

When we run rake test we see:

    .
    failure: 0 tests, on:
    #
    F

    Failures:
    
      1) Peano should succ(n) > n
         Failure/Error: n.succ.should > n
         NoMethodError:
           undefined method `>' for #>
         # ./test/peano_test.rb:96:in `block (3 levels) in '
         # ./test/peano_test.rb:93:in `block (2 levels) in '

We see a decent error message in the final output thanks to RSpec. Just as important – given that this is a test framework using random data – we see a counterexample. Subsequent runs, in this case, would give us different counterexamples. (“In this case” because, since we haven’t defined :> yet, every example is a counterexample!)

Let’s play around a bit, and half-implement :>:

    def > (peano)
      if peano.to_i.even? then
        not (self < peano) and not (self == peano)
      else
        false
      end
    end

Our output then looks like this:

    .
    failure: 0 tests, on:
    #>>>
    F

    Failures:

      1) Peano should succ(n) > n
         Failure/Error: n.succ.should > n
           expected: > #>>>
                got:   #>>>>
           Diff:
           @@ -1,2 +1,2 @@
           -#>>>
           +#>>>>
         # ./test/peano_test.rb:96:in `block (3 levels) in '
         # ./test/peano_test.rb:93:in `block (2 levels) in '

Or, “gosh darn, :> is broken for odd numbers!”

A final note: rantly has some dependencies, but fortunately not that many: rake (naturally), technicalpickles-jeweler, yaml.

In summary, rantly is a simple, easy to use random data generator that works nicely with Test::Unit and is easily extended to use RSpec. It comes with basic generators, and it’s easy to extend the generator support to arbitrary structures.

by
Frank Shearar
on
26/11/11
 
 


5 + = seven

2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us