StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>One way of approaching model based testing is to build a known-good implementation and compare against that. One of the best strategies for doing this is to use a higher-level language so that a short and concise implementation is easy to read and reason about.</p> <p>I've had great success using <a href="http://www.haskell.org" rel="nofollow">Haskell</a> and <a href="http://en.wikipedia.org/wiki/QuickCheck" rel="nofollow">QuickCheck</a> to do model-based testing of a query evaluation. I represented the query language as a Haskell data type, and then simply made this an instance of <code>Arbitrary</code>. Once that was done, QuickCheck could generate infinite numbers of queries, which I could then evaluate against both the server under test and my model implementation.</p> <p><a href="http://www.scala-lang.org/" rel="nofollow">Scala</a> and <a href="http://code.google.com/p/scalacheck/" rel="nofollow">Scala-Check</a> might allow you to do the same sort of thing on the JVM.</p> <p>As a quick example of how this might be used, here's a quick and dirty example:</p> <pre><code>import Test.QuickCheck import Control.Monad (liftM,liftM2) -- An simple expression data type. -- Expression is either (Add X Y, Minus X Y, Multiply X Y or just a plain value) -- where X and Y are further expressions data Expression = Add Expression Expression | Minus Expression Expression | Multiply Expression Expression | Value Int deriving Show -- This is my model-based approach, high level evaluate :: Expression -> Int evaluate (Value x) = x evaluate (Add lhs rhs) = evaluate lhs + evaluate rhs evaluate (Minus lhs rhs) = evaluate lhs - evaluate rhs evaluate (Multiply lhs rhs) = evaluate lhs * evaluate rhs </code></pre> <p>After this I have a known good implementation of a simple model. Just by inspecting the code I can be reasonable sure that this does what I think it should do. Now I can use <code>Arbitrary</code> to generate arbitrarily complicated expressions. The weightings below indicate the frequency (so 70 percent of the generated expressions will be <code>Value</code> and 10 percent will be arbitrary <code>Add</code> statements and so on )</p> <pre><code>instance Arbitrary Expression where arbitrary = frequency [ (70,liftM Value arbitrary) , (10,liftM2 Add arbitrary arbitrary) , (10,liftM2 Minus arbitrary arbitrary) , (10,liftM2 Multiply arbitrary arbitrary) ] </code></pre> <p>This definition gives QuickCheck the power to create me arbitrary expressions. For example:</p> <pre><code>> sample (arbitrary :: Gen Expression) Multiply (Value 1) (Value 0) Value 1 Add (Value (-3)) (Add (Value (-1)) (Minus (Value 3) (Value 0))) Value (-4) Value (-19) Add (Value (-49)) (Value (-55)) Minus (Value 50) (Value 103) Value 210 Value (-172) Value (-907) Value (-2703) </code></pre> <p>I think this is already pretty cool (but maybe I'm just weird). The real powerful bit is that now I can specify an invariant that holds over all generated values</p> <pre><code>prop_testAgainstModel :: Expression -> Bool prop_testAgainstModel expr = evaluate expr == evaluate' expr </code></pre> <p>Where we assume that <code>evaluate'</code> is my super-complicated way of evaluating things that I want to verify against my model. QuickCheck will then generate lots of arbitrary expressions and try to disprove my assertion.</p> <p>Here's an example run against one of my attempts to produce a more complicate evaluate function.</p> <pre><code>> quickCheck prop_testAgainstModel *** Failed! Falsifiable (after 28 tests): Multiply (Value (-53758)) (Value 125360) </code></pre> <p>I used this approach to test an external server written in C++. The Haskell model implementation was maybe a few 100 lines of code and I could compare this against the results of the web service with not much more code than that above.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload