Checking Squeak Quickly
The good fellows in Haskell land came up with a nice idea one day: instead of relying on a programmer writing well-thought out tests, with test data designed to flush out edge cases, they realised that people aren’t very good at finding bugs in their own code. The real world is too random, too crazy, to leave us alone. Things break in production for reasons we would never anticipate. So why don’t we, as part of our testing process, throw some randomness at our tests?
So that’s just what QuickCheck does. You specify a property – something you hold to be true for all your Foos – and QuickCheck will test your property by generating test cases. If it finds a counterexample, it figures out a minimal version of that counterexample and prints it out.
Good ideas tend to spread: JUnit 4.0 has
So why shouldn’t Smalltalkers also have some fun?
We’re going to extend SUnit to support theories, and implement a fairly general generator framework.
SUnit finds its test cases through reflection – if a
TheoryTest >> aPassingTheoryPasses: anObject
self assert: anObject == anObject.
This theory indicates a property shared by all trees, and shows how we declare that we want a particular kind of data:
TreeTheories >> aTreeHeightIsNeverMoreThanItsNumberOfNodes: aTree
self assert: aTree height <= aTree size.
On to how to get our generated test cases. Smalltalk is dynamically typed, so given some theory, it's a lot of work to figure out the type that that theory wants. (Or, it's hard to type a method's parameter.) In Haskell, QuickCheck can leverage the type system to do most of the work - type inference (unification) will figure out what instance of the Arbitrary typeclass to use. So we'll settle for one of two options. <theoryTaking: #SomeClassName> means "this theory expects SomeClassNames passed to it", and <theory> will mean "this theory expects anything".
Putting random data into a test case raises an issue though. You write your theory. The runner finds some counterexample, and falsifies your theory. What value broke your test? Oh, dear. So, two things: we sometimes want to generate non-random data, at the least so that we can build the theory-running infrastructure, and we want to remember the counterexample.
The first of these problems, together with wanting to generate different kinds of data (integers, trees, etc.) sounds a lot like double dispatch. OK, let's do that then:Object >> dataFrom: aDataGenerator
^ aDataGenerator objectData.
Boolean >> dataFrom: aDataGenerator
^ aDataGenerator booleanData.
Integer >> dataFrom: aDataGenerator
^ aDataGenerator integerData.
TrivialDataGenerator >> booleanData
TrivialDataGenerator >> integerData
TrivialDataGenerator >> objectData
Obviously we'd want more kinds of data, and we'd want different generators too. What we have here is sufficient for our initial explorations.
On to the second problem, recording our counterexample. First, some SUnit background. The SUnitbuilds a suite ofTestRunners, Commands representing the execution of a single test. Tests, as mentioned above, are methods on aTestCase, so aTestCaserepresents two things - a collection of tests, and the desire to run one of those tests in aTestCase. ATestRunnerexecutes theTestRunnerCommand by sending it theTestCasemessage which, in turn, results in a self-sent#runCase.performTestsimply hides away the details of running a particular test case whileperformTestmanages the execution: setting up and tearing fixtures, and the like.runCase
Our, in the interests of not being surprising, will support both the usual SUnit style tests, as well as theories. Since the former are nullary message sends while the latter are unary, we'll need to distinguish between the two. Let's add a query methodTheoryTestCase:#runningATheoryTestCase subclass: #TheoryTestCase
TheoryTestCase >> runningATheory
^ testSelector numArgs = 1.
is just the name of the test/theory.testSelectorTheoryTestCase >> runCase
"If counterExample is NotAssigned, then we're running the test for
the first time. Otherwise, it contains the value of a
counterexample to our theory. Run the test using this value."
| prototype |
self runningATheory ifFalse: [^ super runCase].
(currentExample = NotAssigned)
prototype := self makeTestPrototype.
1 to: testRunSize do: [:i |
currentExample := prototype dataFrom: generator.
"A TestFailure, or a timeout, will break out of the loop,
storing the last used value."
on: TestFailure do: [:e |
ifFalse: [super runCase]
TheoryTestCase >> performTest
^ (self runningATheory)
ifTrue: [self performTest: currentExample]
ifFalse: [super performTest]
In particular, when a test is rerun - we have a counterexample to our theory - we will executewhich will callsuper runCase... which will callself performTest.self performTest: currentExample
Two extra details:makes an instance of whatever type the theory takes, and #generateTestCase:withCounterexample: adds a new method to the#makeTestPrototype. So given a theory likeTheoryTestCaseTheoryTest >> aFailingTheoryFails: anObject
self deny: anObject == anObject.
run with our, we will see the theory fail. When we try rerun that test, theTrivialDataGeneratoris set to the datum that falsifies the theory, and we can debug as per usual. Too, thecounterExamplehas a record of the failure:TheoryTestCaseTheoryTest >> testAFailingTheoryFailsWith1
"A test case auto-generated by SqueakCheck."
self aFailingTheoryFails: 1
Having the environment's compiler readily available can be very handy!
And, as always, the load script:Installer ss
"If you'd like to see more complex examples, run the below:"