I recently ran into the need for testing the behaviour of a parser of a modelling language. The parser processes a number of model descriptions in gem files, as well as local definitions. Until recently, the parser would process the gems in an arbitrary order. However the language while ostensibly declarative, isn’t, because of a huge restructuring of the parser. As a result, some bugs lurk in the changed code. These bugs, because of the arbitrary processing order of the model gems, manifest on some machines and not others. Forcing an ordering on the gem processing masks the underlying issues, even while letting the users of the parser to get on with their lives.
What to do? Random testing to the rescue! I cast around for Ruby ports of QuickCheck, and found two: rushcheck and rantly. Rushcheck hasn’t been wrapped up in a gem, so I decided to take rantly for a spin.
The basic structure of a rantly test is:
In other words,
property_of generates some data, and the results are fed into
check, where you describe the property you’re checking.
So, for example, a full spec might be
This example comes from a basic Peano number library I wrote for the purposes of playing with rantly. (I’m really not kidding about “basic”: I’ve intentionally limited the Peano numbers to the range [0, 1000] because I’m lazy.
Rantly supports Test::Unit, so you can always just subclass
Test::Unit::TestCase and write your test as per normal with
assert_equals and friends. Since I like RSpec I had to add a little helper:
rantly supplies a number of basic generators – integers, ranged integers, strings, booleans, chars, etc. – as well as various combinators, and scoped settings. Need an array of between 3 and 5 Peano numbers? No problem:
rantly doesn’t use a polymorphic method for data generation, unlike QuickCheck’s use of typeclasses. It’s hardly difficult, of course, to roll your own such thing:
Most importantly though, a test framework’s only as good as its output. If your assertions don’t result in decent error messages, you might as well not bother. So let’s say we have defined
PNumbers. We would obviously also like
:>. We write up a property (first!):
When we run
rake test we see:
. failure: 0 tests, on: #
F Failures: 1) Peano should succ(n) > n Failure/Error: n.succ.should > n NoMethodError: undefined method `>' for # > # ./test/peano_test.rb:96:in `block (3 levels) in ' # ./test/peano_test.rb:93:in `block (2 levels) in '
We see a decent error message in the final output thanks to RSpec. Just as important – given that this is a test framework using random data – we see a counterexample. Subsequent runs, in this case, would give us different counterexamples. (“In this case” because, since we haven’t defined
:> yet, every example is a counterexample!)
Let’s play around a bit, and half-implement
Our output then looks like this:
. failure: 0 tests, on: #
>>> F Failures: 1) Peano should succ(n) > n Failure/Error: n.succ.should > n expected: > # >>> got: # >>>> Diff: @@ -1,2 +1,2 @@ -# >>> +# >>>> # ./test/peano_test.rb:96:in `block (3 levels) in ' # ./test/peano_test.rb:93:in `block (2 levels) in '
Or, “gosh darn,
:> is broken for odd numbers!”
A final note: rantly has some dependencies, but fortunately not that many: rake (naturally), technicalpickles-jeweler, yaml.
In summary, rantly is a simple, easy to use random data generator that works nicely with Test::Unit and is easily extended to use RSpec. It comes with basic generators, and it’s easy to extend the generator support to arbitrary structures.