A few months back I hinted at the possibility of generative testing being put to use here at Base. “Generative testing” being the idea of writing code to generate tests. The solution discussed here, in particular, allows us to be creative about the sequence and timing of our tests. Let’s get to it!

Typically when writing an automated test for the web it will be linear – its steps executed in the same order each time. While this may be adequate to detect regressions there is a limited chance that the tests will be able to uncover any new bugs. On the other end of the spectrum from linearity would be a randomised approach, embodied by tools such as Android’s Monkey which can cause some interesting crashes as it bashes your UI with events. Occupying the middle ground is the model-based approach: we know our system well enough to know which paths are allowed and what is the likelyhood that they occur. We can use this knowledge to build a model of our application and generate test scenarios based on it.

Perhaps, we’d like to test the performance of our application in a more realistic manner or we’re interested in looking for race conditions in asynchronous code. In both cases we can achieve a lot by manipulating the order and timing of test steps. It’s worth noting that even a very simple model can cover a lot of scenarios.

Building your model

The implementation used here is Craig Andera’s causatum. As the readme desribes it, causatum is “A Clojure library designed to generate streams of timed events based on stochastic state machines.” The first thing we’ll do is define our model.

(ns flow-streamer.models)

(defmulti model (fn [variant] variant))

(defmethod model "commerce" [_]
  {:graph {:home    [{:home    {:weight 3
                                :delay [:constant 2]}
                      :product {:weight 1
                                :delay [:constant 3]}
                      :gone    {:weight 10}}]
           :product [{:home    {:weight 1
                                :delay [:random 10]}
                      :cart    {:weight 3
                                :delay [:constant 4]}
                      :gone    {:weight 2}}]
           :cart    [{:home    {:weight 4
                                :delay [:constant 17]}
                      :gone    {:weight 1}}]}
   :delay-ops {:constant (fn [_ t] t)
               :random   (fn [_ t] (rand t))}
   :initial-graph-node :home})

This is a simplified version of the example given in causatum’s readme. As we walk through the model described in the :graph key, we’ll pick one event at random based on the given weights. The:delay-ops functions define the timing of events and we’ll use the :initial-graph-node as the first event.

Here’s how to generate events from the model:

(ns flow-streamer.streamer
  (:require [causatum.event-streams :as es]))

(defn initial-stream-state [model]
  [{:state (:initial-graph-node model) :rtime 0}])

(defn model->stream [model]
  (es/event-stream model (initial-stream-state model)))

Let’s look at some example outputs:

({:state :home, :rtime 0}
 {:delay [:constant 2],:weight 3, :state :home, :rtime 2}
 {:delay [:constant 2], :weight 10, :state :gone, :rtime 2})

({:state :home, :rtime 0}
 {:weight 10, :state :gone, :rtime 0})

({:state :home, :rtime 0}
 {:delay [:constant 3], :weight 1, :state :product, :rtime 3}
 {:delay [:constant 4], :weight 3, :state :cart, :rtime 7}
 {:delay [:constant 17], :weight 4, :state :home, :rtime 24}
 {:delay [:constant 17], :weight 10, :state :gone, :rtime 24})

Exposing event streams via HTTP

Great, we have a certain degree of variability in the order and timing of events which we can use to design our tests. But what if your test suite isn’t using Clojure? Maybe you already have a suite of Webdriver tests with laid out PageObjects that you’d like to use. Let’s see how we can expose such an event stream via HTTP for use elsewhere.

First, let’s wrap our code in a HTTP API using metosin/compojure-api

(ns flow-streamer.handler
  (:require [compojure.api.sweet :refer :all]
            [ring.util.http-response :refer :all]
            [schema.core :as s]
            [flow-streamer.streamer :as streamer]
            [flow-streamer.models :as models]))

(s/defschema FlowStream {:stream [{:state s/Keyword
                                   :rtime s/Num
                                   (s/optional-key :delay) s/Any
                                   (s/optional-key :weight) s/Num}]})

(defapi app
  (swagger-ui)
  (swagger-docs
    {:info {:title "flow-streamer"
            :description "Generates streams of events from a model"}})
  (context* "/flows" []
    (GET* "/:model-name" [model-name]
      :return FlowStream
      :summary "returns an event stream from the chosen model"
      (let [model (models/model model-name)
            stream (streamer/model->stream model)]
        (ok {:stream stream})))))

Here’s the project.clj for the app:

(defproject flow-streamer "0.1.0-SNAPSHOT"
  :description "Generates streams of events from a causatum model"
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [clj-time "0.9.0"]
                 [metosin/compojure-api "0.22.0"]
                 [org.craigandera/causatum "0.3.0"]]
  :ring {:handler flow-streamer.handler/app}
  :profiles {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
                                  [cheshire "5.3.1"]
                                  [ring-mock "0.1.5"]
                                  [midje "1.6.3"]]
                   :plugins [[lein-ring "0.9.6"]
                             [lein-midje "3.1.3"]]}})

To run the web app make sure you have Leiningen installed and run:

lein deps

lein ring server-headless

This will start on port 3000. Our API exposes swagger docs so go to http://localhost:3000 and try out some requests. We’ll get our event streams in the JSON format.

swagger ui

Swagger UI

 

Generating RSpec tests

OK, now it’s time to use the output to generate RSpec tests in Ruby. This may sound specific but the principles should be the same whatever technology or test runner you fancy. We’ll fetch our event stream and turn it into a spec using an erb template.

require "erb"
require "httparty"

class FlowStream(template_path, model_name)
  def call
    @steps = steps
    render
  end

  private
  def render
    ERB.new(template).result(binding)
  end

  def steps
    raw_flow_stream.
      fetch("stream").
      map do |step|
        {name: step.fetch("state"),
         delay: step.try(:[], "delay").try(:last)}
      end
  end

  def raw_flow_stream
     HTTParty.get("http://localhost:3000/flows/commerce")
  end

  def template
    File.read(template_path)
  end
end

In its simplest form our template could look like this:

require "spec_helper"
require_rel 'support'

describe "Generative test" do
  <% @steps.each do |step| %>
    include_examples "<%= step[:name] %>", <%= step[:delay] || 0 %>
  <% end %>
end

For each part of the flow we insert a shared example with the delay in seconds as an argument. An example test may end up looking like this:

require "spec_helper"
require_rel 'support'

describe "Generative test" do

  include_examples "home", 0

  include_examples "product", 3

  include_examples "cart", 4

  include_examples "process-order", 4

  include_examples "home", 17

  include_examples "gone", 17

end

The fact that we use files as input for RSpec means that we can easily re-run a failing scenario with the same parameters.

Coda

That’s it! There is far more to causatum then what I desribed here so I encourage you to look at the readme and figure out your own use-cases. This example is suitable for controlling a single thread of test execution. To see how generative testing can be taken to the next level check out Michael Nygard’s Simulant example. Over and out.


Cover photo by darkday licensed with Creative Commons Attribution-Noncommercial-No Derivative Works 2.0 Generic License

Posted by

Wojtek Franke

Share this article