Randomly testing Ruby

I recently ran into the need for testing the behaviour of a parser of a modelling language. The parser processes a number of model descriptions in gem files, as well as local definitions. Until recently, the parser would process the gems in an arbitrary order. However the language while ostensibly declarative, isn’t, because of a huge restructuring of the parser. As a result, some bugs lurk in the changed code. These bugs, because of the arbitrary processing order of the model gems, manifest on some machines and not others. Forcing an ordering on the gem processing masks the underlying issues, even while letting the users of the parser to get on with their lives.

What to do? Random testing to the rescue! I cast around for Ruby ports of QuickCheck, and found two: rushcheck and rantly. Rushcheck hasn’t been wrapped up in a gem, so I decided to take rantly for a spin.

The basic structure of a rantly test is:

    property_of {

      my_special_generator

    }.check { |generator, values|

      my_special_property(generator, values)

    }

In other words, property_of generates some data, and the results are fed into check , where you describe the property you’re checking.

So, for example, a full spec might be

    class Rantly

      def peano(limit = nil)

        limit = 0..Peano::MAX_INT if limit.nil?

        Peano.from_i(integer(limit))

      end

    end



    module Peano

      describe "Peano" do

        it "should 0 < n" do

          property_of {

            peano(1..100)

          }.check { |n|

            Peano.zero.should < n

          }

        end

      end

    end

This example comes from a basic Peano number library I wrote for the purposes of playing with rantly. (I’m really not kidding about “basic”: I’ve intentionally limited the Peano numbers to the range [0, 1000] because I’m lazy.

Rantly supports Test::Unit, so you can always just subclass Test::Unit::TestCase and write your test as per normal with assert_equals and friends. Since I like RSpec I had to add a little helper:

    class RSpec::Core::ExampleGroup

      def property_of(&block)

        Rantly::Property.new(block)

      end

    end

rantly supplies a number of basic generators – integers, ranged integers, strings, booleans, chars, etc. – as well as various combinators, and scoped settings. Need an array of between 3 and 5 Peano numbers? No problem:

    # When you want to share the size between multiple generators ...

    sized(integer(3..5)) {

      array { peano }

    }



    # ... or when you don't need to.

    array(integer(3..5)) { peano }

rantly doesn’t use a polymorphic method for data generation, unlike QuickCheck’s use of typeclasses. It’s hardly difficult, of course, to roll your own such thing:

    class PNumber

      def self.generator

        Peano.from_i(Rantly.integer(0..Peano::MAX_INT))

      end

    end

Most importantly though, a test framework’s only as good as its output. If your assertions don’t result in decent error messages, you might as well not bother. So let’s say we have defined :== and :< for PNumber s. We would obviously also like :> . We write up a property (first!):

    it "should succ(n) > n" do

      property_of {

        PNumber.generator(0..3)

      }.check {|n|

        n.succ.should > n

      }

    end

When we run rake test we see:

    .
    failure: 0 tests, on:
    #
    F


Failures:

  1) Peano should succ(n) > n
     Failure/Error: n.succ.should > n
     NoMethodError:
       undefined method `>' for #<Succ #<Zero>>
     # ./test/peano_test.rb:96:in `block (3 levels) in <module:Peano>'
     # ./test/peano_test.rb:93:in `block (2 levels) in <module:Peano>'

We see a decent error message in the final output thanks to RSpec. Just as important – given that this is a test framework using random data – we see a counterexample. Subsequent runs, in this case, would give us different counterexamples. (“In this case” because, since we haven’t defined < no_cc="true"code>:> yet, every example is a counterexample!)

Let's play around a bit, and half-implement :> :

    def > (peano)

      if peano.to_i.even? then

        not (self < peano) and not (self == peano)

      else

        false

      end

    end

Our output then looks like this:

    .
    failure: 0 tests, on:
    #>>>
    F

    Failures:

      1) Peano should succ(n) > n
         Failure/Error: n.succ.should > n
           expected: > #>>>
                got:   #>>>>
           Diff:
           @@ -1,2 +1,2 @@
           -#>>>
           +#>>>>
         # ./test/peano_test.rb:96:in `block (3 levels) in '
         # ./test/peano_test.rb:93:in `block (2 levels) in '

Or, "gosh darn, :> is broken for odd numbers!"

A final note: rantly has some dependencies, but fortunately not that many: rake (naturally), technicalpickles-jeweler, yaml.

In summary, rantly is a simple, easy to use random data generator that works nicely with Test::Unit and is easily extended to use RSpec. It comes with basic generators, and it's easy to extend the generator support to arbitrary structures.