Or: How I Learned to Stop Worrying and Love Testing

Introduction

This document deals principally with initial set up and first use of the Cucumber-Rails gem, that uses Cucumber, a behaviour driven development (BDD) tool, for designing and implementing a project using the Ruby on Rails (RoR) framework. Discussion of Behaviour Driven, Test Driven (TDD), and Panic Driven Development (SNAFU aka Cowboy Coding) can be found elsewhere. Details regarding installing the Cucumber gem and its recommended support tools for RoR are found on this wiki under the heading Ruby on Rails. The article Getting Started with Rails is a useful introductory tutorial dealing with Rails and Cucumber.

Since the original version of this article appeared, Cucumber has undergone repeated revisions and refactorings. Among these was the sensible decision to move portions of the implementation specific to particular frameworks into their own gems. Consequently, installing Cucumber for a framework now frequently starts with installing the specific framework Cucumber gem, which in turn pulls in the core Cucumber gem as a dependency.

Note that in this document I often use the terms testing and test where BDD practictioners prefer the terms behaviour and expectation. When I write test I am in fact discussing expressing and verifying expected behaviour.

Where to Start?

Feature: Design and Build a Ruby on Rails web app using Behaviour Driven Development (BDD)
In order to reduce rework and produce a web app at low cost and high speed
A developer 
Should employ a BDD methodology and agile tools

Scenario: Cucumber should be installed and configured
Given I have installed the gem named "rails"
  And I have installed the gem named "cucumber-rails"
  And I have generated a RoR project named "my_project"
  And the present working directory is the Rails root directory of "my_project"
  And I have created the file cucumber.yml in the root directory
  And the file cucumber.yml contains a valid default profile

When I run "script/generate cucumber"

Then I should create the directory ./features
  And I should create the file ./features/features.feature
  And I should create the directory ./features/step_definitions
  And I should create the file ./features/step_definitions/web_steps.rb
  And I should create the directory ./features/support
  And I should create the file ./features/support/env.rb
  And I should create the file ./features/support/paths.rb
  And I should create the file ./lib/tasks/cucumber.rake
  And I should create the file ./config/environments/environment cucumber.rb
  And I should modify ./config/database.yml
  .  .  .

The foregoing gives a sample of the form that feature files often take. The statements, called feature steps, make up the user interface to Cucumber testing. These here are written in the Imperative Style for illustrative purposes. In practice, all those Then/And statements likely would be subsumed into one simple Declarative statement; for example: I should create the Cucumber environment, and the messy details of just what a Cucumber environment comprises placed in the step definitions instead. We will return to that topic later. For the moment we deal with the logical arrangement of Cucumber files within the context of a RoR project.

The root level of the archetypal RoR project directory tree looks like this:

MyProject
|-- README
|-- Rakefile
|-- app
|-- config
|-- db
|-- doc
|-- lib
|-- log
|-- public
|-- script
|-- test
|-- tmp
`-- vendor

Running script/generate cucumber adds this layout to the existing structure:

|-- features
|   |-- step_definitions
|   |   `-- web_steps.rb
|   `-- support
|       |-- env.rb
|       `-- paths.rb

We are now ready to begin testing with Cucumber.

Where do I put Tests?

Cucumber divides testing into two parts, the outward facing feature steps and the inward facing step definitions. As discussed elsewhere, features are descriptions of desired outcomes (Then) following upon specific events (When) under predefined conditions (Given). They are typically used in conjunction with end-user input and, in some cases, may be entirely under end-user (in the form of a domain expert) control. Feature files are given the extension .feature.

A source of potential confusion is that the term steps, when used loosely, has two, closely related but vitally distinct, meanings depending on context. Inside feature.feature files, steps are the textual descriptions which form the body of a scenario. These are prefaced with Given, When, Then, and And (note as well that the capitalization of these four keywords/method names is significant). Inside a step_definitions.rb file, steps (which strictly speaking should always be called step definitions) refers to the matcher methods, given exactly the same names (Given, When, or Then, but not And), each provided with a matcher regexp that corresponds to one or more feature steps. Note that the method name does NOT form part of the matcher. A Given feature clause can match a When step definition matcher. Over time, clauses from features have come to be referred to simply as features while steps now refers almost exclusively to step-matchers

Step definitions, keyed by their snippets of text from the feature scenario statements, invoke blocks of Ruby and Rails code that usually contains assertion statements from whatever test system you have installed. Given that Cucumber evolved out of RSpec stories it is unsurprising that the Cucumber generator once assumed that RSpec was available. This has long since ceased to be the case. What Cucumber does now is detect if the rspec and rspec-rails gems are installed. If so then the rails generator builds the environment files to suit and if not then it ignores rspec and configures for test-unit instead. Further, the availablity of Cucumber generator options is increasing over time. To see what is available in the version of Cucumber-Rails that you have installed use the command: script/generate cucumber --help.

As detailed above, the default features directory tree is fairly shallow. One can put every feature into a single file in the features directory and every step in a single file in the steps_definition directory (or even in the features directory itself) if one so chooses. One can choose to have one or more feature files for each feature together with one or more step files for each feature file; or any combination thereof. However, Cucumber is programmed with the flexibility to support a much more expressive directory structure. For instance:

|-- features
|   |-- entities
|   |   |-- entity.feature
|   |   `-- step_definitions
|   |       |-- anything.rb
|   |       `-- entity_steps.rb
|   |-- locations
|   |   |-- location.feature
|   |   `-- step_definitions
|   |       `-- location_steps.rb
|   |-- sites
|   |   `-- step_definitions
|   |-- step_definitions
|   |   `-- webrat_steps.rb
|   `-- support
|       `-- env.rb

In this case the bland initial setup has been divided into sub-directories informed by model-centric testing. This could equally well have been broken up in to model/controller/view hierarchies:

|-- features
|   |-- models
|   |   |-- entities
|   |   |   |-- entity.feature
|   |   |   `-- step_definitions
|   |   |       |-- anything.rb
|   |   |       `-- entity_steps.rb
|   |-- views
|   |   |   |-- entity_new
|   |   |   `-- step_definitions
|   |   |       `-- entity_new_steps.rb
|   |-- step_definitions
|   |   `-- webrat_steps.rb
|   `-- support
|       `-- env.rb

Consider, however, that the Cucumber feature scaffold generator will generate the files it produces in conformance with the default layout, with the manage_frooble.feature file placed in ./features and the frooble_steps.rb file in ./features/step_definitions. Also be aware that, regardless of the directory structure employed, Cucumber effectively flattens the features directory tree when running tests. By this I mean that anything ending in .rb under the start point for a Cucumber feature run is searched for feature matches. So that a step contained in features/models/entities/step_definitions/anything.rb can be used in a feature file contained in features/views/entity_new. It is also worth noting that step files can be called anything so long as they end in .rb.

How do I Write Tests?

Constructing ones first tests, or features as BDD purists prefer, is often accompanied by what can only be described as writer’s block. While detailed discussion of feature writing and step construction are provided elsewhere (see Given-When-Then and Telling a Good Story), perhaps the easiest thing to do for the first time tester/behaviourist is to use Cucumber’s built-in generator to create a feature scaffold and then modify it.

script/generate feature Frooble name color description
      exists  features/step_definitions
      create  features/manage_froobles.feature
      create  features/step_definitions/frooble_steps.rb

While scaffolding provides a comforting illusion of progress through its voluminous production of template code, in practice it proves important to write each test/feature step one at a time. After each new feature step is added then you should immediately create the corresponding step definition method. After creating your new test/step definition you must prove to yourself that it fails by running it against the, as yet, non-existant application code. Then, and only then, should you write the least application code that gets your test/step definition to pass. Once this cycle is complete then move on to the next feature step.

It is often tempting to just skip ahead with the analysis and complete as many scenarios and feature steps as one can imagine. In some cases limited access to domain experts and end users may require that many feature steps be completed long before coding the associated step definitions is undertaken. Nonetheless, in the long run, it has proven best practice to write feature steps and step definitions incrementally, using the absolute minimum of code; and then immediately implement the new feature step requirement in application code, also using the minimum code to satisfy it.

This is a hard discipline to accept but the value with this approach is that you will rarely (never) have untested code anywhere in your application and your application will only have code that satisfies required features. The benefit to that is that you can face design changes with complete equanimity, secure in the knowledge that if unanticipated changes break anything else then you will know of the side-effect immeditely after running your test suite.

If you do anticipate feature steps then omitting any matcher for them in the step definitions files causes these steps to be reported as missing by Cucumber, which helpfully provides a suggested method and argument to implement. Stub step matchers have an explicit pending method avaiable to designate defined but pending/undefined step definitions.

A single Feature is typically contained in its own file (ending in .feature). Each Feature usually consists of multiple Scenarios. Each Scenario consists of three sections, Given, When and Then. Each section consists of one or more clauses (steps) that are used to match test step definitions. The conventional arrangement is:

Feature: Some terse yet descriptive text of what is desired
In order that some business value is realized
An actor with some explicit system role
Should obtain some beneficial outcome which furthers the goal
To Increase Revenue | Reduce Costs | Protect Revenue  (pick one)

  Scenario:   Some determinable business situation
      Given some condition to meet
         And some other condition to meet
       When some action by the actor
         And some other action
         And yet another action
       Then some testable outcome is achieved
         And something else we can check happens too

  Scenario:  A different situation
      ...

For Cucumber features the key words used here are Feature, Scenario, Given, When, Then, and And. Feature is used to provide identification of the test group when results are reported. Scenario is used in the same fashion. Given, When, Then, and And are all Cucumber methods that take as their argument the string that follows. These are the steps that Cucumber will report as passing, failing or pending based on the results of the corresponding step matchers in the step_definitions.rb files. These four methods are also all equivalent.

def Given(name)
  create_step('Given', name, *caller[0].split(':')[1].to_i)
end
def When(name)
  create_step('When', name, *caller[0].split(':')[1].to_i)
end
def Then(name)
  create_step('Then', name, *caller[0].split(':')[1].to_i)
end
def And(name)
  create_step('And', name, *caller[0].split(':')[1].to_i)
end

The string argument from each of these methods is compared against all the matchers contained in the step_definitions.rb files. A step definitions matcher looks much like this:

Given /there are (\d+) froobles/ do |n|
  Frooble.transaction do
    Frooble.destroy_all
    n.to_i.times do |n|
      Frooble.create! :name => "Frooble #{n}"
    end
  end
end

The significant thing here is that the method (Given) takes as its argument a regexp bounded by /. Among other things, that means that all the arguments are received as string values. Thus n.to_i.times and not simply n.times.

In the features provided above we had the statement: And some other action. This could be matched by any of the following step definition matchers if present in any step_definitions.rb file found under the features root directory.

Given /some other action/ do
When /some (.*) action/ do |match|
Then /(*.) other action/ do |match|
Given /(.*) other (.*)/ do |first,second|

And /some other act.*/ do 

The exception occurs because And is not defined as a steps definition method, but only as a feature step method. The step definition match depends only upon the pattern following the Given/When/Then method and not upon the step method itself. I have therefore adopted the practice of only using When /I have a match/ do in my step definitions files as When has a more natural appearance for a matcher.

If more than one matcher in all of the step definitions files matches a feature step then Cucumber complains that it found multiple step definition matches for that step and forces you to distinguish them. You can force Cucumber to make a decision instead by passing the switch --guess on the command line.

It is considered better form by some to surround with double quotation marks, " ", elements in the feature step clauses that are meant to be taken as values. This is only a convention and one not followed by many. However, if you choose to follow this road then you must adjust your step definition matchers accordingly. For example:

Given Some determinable "business" situation

Given /determinable "(.*)" situation/ do |s|

Finally, you can have step definitions call other step definitions, including those contained in other step definitions files. For example:

When /some "(.*)" action/  do |a|
   ...
end

When /in an invoiced non-shipped situation/ do
  Given "some \"invoiced\" action" 
  Then "some \"non-shipped\" action"
  ...
end

If one step definition calls another step definition then the matcher argument to the called Given/When/Then method must be enclosed with string delimiters. Because of this, if you have adopted the practice of demarcating parameter values present in feature steps with double quotation marks, you must escape these quotation marks when calling another definition_steps.rb matcher from inside a step_definitions.rb file. You must take care not to include the quote marks in the step_definitions parameter matchers, for "(.*)" is not the same as (.*) or (".*"). If you use quote delimited values in the .feature file steps and do not account for them in the corresponding step_definition.rb matcher regexp then you will obtain variables that contain leading and trailing quotes as part of their value.

Scenario: Quotes surround value elements
  Given some "required" action

# step_definitions
When /some (.*) action/ do |a|
  a => "required"

When /some "(.*)" action/ do |a|
  a => required

In a manner similar to my convention of using only using the When method for step definition matchers, since When, Given, and Then are all equivalent inside the step definitions file I have adopted the convention of calling other step definitions within matchers using the Then method, thus:

When /my matcher named (.*)/ do |match|
  Then "my other matcher named \"#{match}\""
end

The choice of method has no impact on the result but, When and Then used in this fashion seem to me to read more naturally.

If all your feature’s scenarios share the same ‘setup’ feature steps then Cucumber provides the Background section. Steps contained within a Background section are run before each of the scenarios.

Feature: .  .  .
  Background: .  .  .
  Scenario: .  .  .

What Way do I Run the Tests?

Unless you are knowledgeable enough that you use mocks and stubs then I consider it best to begin with creating a Rails migration file for the model you are testing ( or expressing features for ) followed by:

rake db:migrate
rake db:test:prepare

However, there is an important environmental consideration to keep in mind when using an actual database for testing: Cucumber, by default, uses database transactions and these transactions are rolled back after each scenario. This makes out-of-process testing problematic and may result in some unanticipated outcomes under certain scenarios. Transactions can be turned off, but then your features become responsible for ensuring that the database is in a condition suitable for testing. Cucumber provides hooks to accomplish this and the gem Database-Cleaner is configured in support/env.rb to assist.

Testing with Cucumber can be performed in several fashions. Be aware that rake cucumber, cucumber features, and autotest with ENV AUTOFEATURE=true do not necessarily produce the same results given the same tests.

Running rake cucumber from the command line provides the simplest, if not the speediest, method to run Cucumber tests. The rake script provided with cucumber performs much of the background magic required to get the test database and requisite libraries properly loaded. The Cucumber Rake task now recognises the @wip tag, so rake cucumber:wip will run only those scenarios tagged with @wip. For example, given a feature file containing:

Feature: .  .  .
   
  Scenario: A

  @wip
  Scenario: B

  Scenario: C

running the command rake cucumber:wip will run the steps contained inside Scenario B only, while running rake cucumber:ok will run the steps within all Scenarios other than B.

Running Cucumber first requires the creation of a cucumber.yml file in the project root directory and that file must contain a default profile (for example: default: --require features --format pretty. When Cucumber is run from the command line it is usually necessary to provide both the directory name containing the root directory of the tree containing feature files and the directory name containing references to the necessary library files. In the typical project cucumber features -r features will normally suffice.

Finally, running autotest with the environment variable AUTOFEATURE=true will run ALL tests, including those in /test and (if present) /spec. As this will load all test fixtures the test database may be left in an indefinite state when Cucumber features are run. It is wise, as always, to write Cucumber steps either so that they do not depend upon an empty database or they place the database in the requisite state.

Anything Else?

The terminology for elements of Behaviour Driven Development differs somewhat from that employed by Test Driven Development. This article, because of the introductory nature of its contents, tends to blur the semantic distinction between these two divergent philosophies.

Realize that tests/assertions/expectations either pass or fail (raise an error) and that fail is NOT the same as false, whereas anything but fail is a pass. When, in RSpec, something.should_be 0 and it is not, then what is returned is an error exception and not a boolean value. In raw Cucumber (pardon the pun) one writes fail if false and not simply false. A little reflection reveals why this is so, since false might be the expected successful outcome of a test and thus not an error. However, the distinction between fail and false escaped my notice until I was confronted with it in an actual test suite.

If you are testing with intent then you should be using the ruby-debug gem. A really neat method to drop into an interactive debugging session inside a Cucumber step definition using ruby-debug was provided by Scott Taylor on the rspec mailing list. Just put these statements inside the step definition at the point that you wish to debug: require 'rubygems'; require 'ruby-debug'; debugger. When that code interrupts then type irb and you open an interactive debugging session wherein you can step forwards and backwards inside the code under test to determine exactly where the breakage is happening. Alternatively, you can add require 'rubygems'; require 'ruby-debug'; to your support/local_env.rb file (see below) and just put debugger wherever you desire it inside any step definition.

Recall that Cucumber is an integration test harness. It is designed to exercise the entire application stack from view down to the database. It is certainly possible to write features and construct step definitions in such a fashion as to conduct unit tests and no great harm will result. However, this is not considered best practice by many and may prove insupportable for very large projects due to the system overhead it causes and the commensurate delay in obtaining test results.

Cucumber-Rails is preconfigured with support for view integration testing using either Capybara (script/generate cucumber —capybara) or Webrat (script/generate cucumber). If you do not know what Capybara or Webrat are or do, and you are doing web application testing, then you should find out more about both.

The Cucumber generator will set up the necessary support files whichever you choose.

Calls for any additional Ruby libraries that your particular testing environment may require are typically be placed in the ./features/support/local_env.rb file. I advise against putting local customization in support/env.rb as it is typically overwritten by script/generate cucumber. As a matter of good practice you should always run script/generate cucumber whenever you install an updated version. You should do the same for rspec as well.

Those of you that have used growl or snarl to provide desktop notifiers from autotest are advised that, as of this writing, Cucumber did not hook into the :red :green notifier capability of autotest; so, no popups when a step fails. However, there exists a project to add a similar functionality to Cucumber. See Cucumber_Growler.

autotest is installed via the ZenTest gem. If you use autotest then take a look at the contents of example_dot_autotest.rb in the ZenTest gem root directory.

Need Help?

The best place to go for help, that I know of, is the Google Cucumber Group.

If you find a bug in Cucumber, or wish a new feature added, then you should open a ticket at Lighthouse for it.

2008 November 28 – J. B. Byrne initial
2010 January 17 – J.B. Byrne revised

Postscript.

A caution, Cucumber is meant to facilitate expressing required behaviours. Indirection and excessive adherence to the principle of DRY, particularly in features, is at variance with both the intent and the major benefit of the tool. Requirement expression in features should remain as self evident to the non-technical and as self contained as possible. Resist the tempation to program features in Cucumber.