Clarify the role of gnatbdd and the driver in the documentation

This commit is contained in:
Emmanuel Briot 2014-06-06 11:40:41 +02:00
parent 77d79c5311
commit dfc490af5a
6 changed files with 251 additions and 158 deletions

87
docs/driver.rst Normal file
View File

@ -0,0 +1,87 @@
***************
The test driver
***************
The test driver is the executable generated by GNATbdd. In fact, there could
be several drivers built, depending on how you chose to run GNATbdd. For
instance, one of the driver might stub part of your application, whereas
another driver would test that same part and stub another part.
Finding features
================
Let's first examine how the driver finds all the features for your application.
.. index:: switches; --ext
Typically, the features are written in text files with a :file:`.feature`
extension. Several editors will automatically provide syntax highlighting with
such an extension. However, you can choose your own extension by using the
:option:`--ext .EXT` switch when you run the driver.
.. index:: switches; --features
By default, the driver will find all the files with a matching extension in
a subdirectory names :file:`features`, within the directory where the driver
is run. You can however specify one or more other directories to check,
by using the :option:`--features DIR` switch one or more times.
The driver will always search recursively in those directories. This lets you
organize the files into subdirectories when you have several of them, and
thus perhaps make it easier to refer to them.
The driver will run the features file in alphabetical order of the full path of
the feature file, by default. This provides a more consistent output when
the driver is run multiple times.
.. index:: switches; --random
However, by using the :option:`--random` switch, the order of features is
randomized. This is a convenient way to check that the various tests are indeed
independent of each other, as they should be to make it possible to run tests
individually.
Running feature
===============
.. index:: switches; --tags
When you run the driver, you can use the switch :option:`--tags` to control
which scenarios should be run. For instance, all scenarios related to '@gui',
or all scenarios not related to '@startup'. You can of course select subsets of
scenarios based on the file names, but tags provide a file-system-agnostic
selection mechanism.
Step timeout
============
.. note::
This will likely require each scenario to be run in its own task. There
will be only one such task, so it doesn't really add constraints on the
user code or step definitions, but it makes debugging slightly more
difficult.
.. note::
Perhaps we can also change the way the driver is generated, rather than do
everything dynamically in the driver itself.
.. index:: switches; --timeout
You can use the :option:`--timeout` switch to specify a maximum time that
steps can take to execute. A test that times out will automatically fail
with an appropriate error message.
Running tests in parallel
=========================
There are multiple modes to run features in parallel. The parallelism is always
between :file:`.feature` file, never between the Scenario of a given file.
When your application is task safe, you can run multiple features in parallel
by running each in its own task (up to a maximum number of tasks of course).
In other cases, the driver can automatically spawn several instances of itself,
each running a single feature in parallel of the others. This is slightly less
efficient, but does not impose task-safety constraints on your application.

View File

@ -12,40 +12,6 @@ meant for human readers, and is not used by the automatic tool.
The features are then further divided into one or more scenarios, which are
sometimes also called "tests". These will be discussed in the next section.
Finding features
================
Let's first examine how GNATbdd finds all the features for your application.
.. index:: switches; --ext
Typically, the features are written in text files with a :file:`.feature`
extension. Several editors will automatically provide syntax highlighting with
such an extension. However, you can choose your own extension by using the
:option:`--ext .EXT` switch when you run GNATbdd.
.. index:: switches; --features
By default, GNATbdd will find all the files with a matching extension in
a subdirectory names :file:`features`, within the directory where GNATbdd
is run. You can however specify one or more other directories to check,
by using the :option:`--features DIR` switch one or more times.
GNATbdd will always search recursively in those directories. This lets you
organize the files into subdirectories when you have several of them, and
thus perhaps make it easier to refer to them.
GNATbdd will run the features file in alphabetical order of the full path of
the feature file, by default. This provides a more consistent output when
GNATbdd is run multiple times.
.. index:: switches; --random
However, by using the :option:`--random` switch, the order of features is
randomized. This is a convenient way to check that the various tests are indeed
independent of each other, as they should be to make it possible to run tests
individually.
Syntax of the features file
===========================
@ -123,20 +89,11 @@ to run subsets of the whole set of scenarios. Here is an example::
The tags of the feature automatically apply to its Scenarios
.. index:: switches; --tags
When you run GNATbdd, you can use the switch :option:`--tags` to control which
scenarios should be run. For instance, all scenarios related to '@gui', or all
scenarios not related to '@startup'. You can of course select subsets of
scenarios based on the file names, but tags provide a file-system-agnostic
selection mechanism.
Other usage of tags could be to identify *slow tests* (with @slow) so that
their timeout is increased.
A tag can also be used to link a scenario to a *high-level requirement* in your
application
application, for instance @HLR-12-2.
Tags can also be used to identify *expected failures* (for instance @xfail), or
*work in progress* (for instance @wip).

View File

@ -13,6 +13,7 @@ GNAT Behavior Driven Development
bdd
features
steps
driver
output
customizing
ideas

View File

@ -2,7 +2,7 @@
Controlling the output
**********************
GNATbdd provides a rich set of options for controlling the output.
The driver provides a rich set of options for controlling the output.
Colors
======
@ -10,7 +10,7 @@ Colors
.. index:: switches; --color
Color output is enabled by default when the terminal supports it. It is used to
highlight passing, failing and skipped tests. GNATbdd attempts to detect
highlight passing, failing and skipped tests. The driver attempts to detect
automatically whether advanced features like colors or moving the cursor are
supported. However, you can also force a specific setting by using one of:
@ -18,14 +18,14 @@ supported. However, you can also force a specific setting by using one of:
Forces the use of colors in the output. This is in particular useful when the
output is redirected to a file which you intend to display in a console later
on. GNATbdd always uses ANSI escape sequences when redirecting to a file,
on. The driver always uses ANSI escape sequences when redirecting to a file,
so on Windows you should install `ANSICON <https://github.com/adoxa/ansicon/>`_.
This is not necessary when you are directly displaying the output in the
console though.
* :option:`--color=no`
Disable the use of colors. GNATbdd will output strings such as "[OK]",
Disable the use of colors. The driver will output strings such as "[OK]",
"[FAILED]" or "[SKIPPED]" instead.
* :option:`--color=auto` (the default)
@ -45,7 +45,7 @@ as possible directly in the console, so that hopefully it is easy to undertand
why a test is failing. In fact, some users might want to see the passing tests
as well, just to make sure the testsuite is moving forward.
However, during an automatic test, all the output can be logged, and we are
However, during an automatic test, all the output can be dropped, and we are
only interested in getting a final count of passing and failing tests.
All these behavior can be controlled with the following command line
@ -53,7 +53,7 @@ switches:
* :option:`--output=quiet`
GNATbdd only displays the final count for each categories of
The driver only displays the final count for each categories of
tests (passed, failed and skipped).
The output therefore looks like::
@ -62,11 +62,11 @@ switches:
* :option:`--output=dots` (the default)
GNATbdd displays a "." for each test that passes, a "F" for each
The driver displays a "." for each test that passes, a "F" for each
test that fails (and will then display a full backtrace for those)
and a "S" for skipped tests.
When all features have run, GNATbdd displays details for all tests
When all features have run, the driver displays details for all tests
that did not pass.
Finally, it displays a summary for the final count for each category.
@ -87,26 +87,40 @@ switches:
* :option:`--output=hide_passed`
In this mode, GNATbdd does not display anything when a scenario passes,
In this mode, the driver does not display anything when a scenario passes,
but displays the same backtrace as above for failed or skipped. One
advantage is that the output of the full traces is done after each
scenario has been run, not as a summary after all of them have run.
* :option:`--output=full`
In this mode, GNATbdd displays output of all scenarios, whether they
In this mode, the driver displays output of all scenarios, whether they
pass or fail. The output looks like the above, but instead of displaying
a ".", GNATbdd outputs the full lists of steps for that scenario.
a ".", the driver outputs the full lists of steps for that scenario.
In this mode, GNATbdd outputs each step before it is being run, thus this
In this mode, the driver outputs each step before it is being run, thus this
allows you to monitor which step is taking long to execute.
Redirecting the output
======================
.. index:: switches; -o
By default, the driver will print its output on the screen. It is possible
though to print to a file by using the :option:`-o FILE` switch.
.. index:: HTML output
If the file ends up with the :file:`.html` extension, the driver will
generate HTML output rather than textual output. This will display nicer
in a web browser.
Log files
=========
.. index:: switches; --log
In addition, GNATbdd can still generate log files which contain output
In addition, the driver can still generate log files which contain output
similar to the above, as well as any output done by your application
on stdout or stderr.
@ -120,20 +134,6 @@ subdirectory of the directory where GNATbdd is run.
Should this be in the root project's object_dir, when GNATbdd is run
with a project in argument ?
Projects
========
.. index:: switches; -P
GNATbdd needs to compile the step definitions your application provides. The
features files themselves are parsed dynamically and do not need compile.
For the compilation to succeed, GNATbdd needs to find where the sources of the
steps are (see the :option:`--steps` switch), as well as all the sources on
which they depend. Such sources are found via a GNAT Project file, which can
be specified with the :option:`-P PROJECT.gpr` switch.
Exit status
===========

View File

@ -2,6 +2,63 @@
Steps definitions
*****************
GNATbdd
=======
GNATbdd is the first tool that you will need to run as part of your testing
framework.
Its role is to aggregate, into one or more executables various pieces of
code:
Part of your application's code
This is the code you intend to test, and all its closure and dependencies.
Step definitions
These are (hopefully short) subprograms that create the link between the
English sentences you wrote in your features files, and the actual code to
execute.
GNATbdd's own library
GNATbdd comes with an extensive library which includes looking for features
files to run, parsing them, actually running them (or a subset of them
depending on command line switches), and an assert library to help you
write your own tests.
Stubs
It is often useful, when running test, to stub a part of your application.
This might be done to simplify the test setup (no need for a remove server
for instance for a distributed application), to speed up the test by
abstracting a slower part of the application, or to help focus the tests.
Since you might want to stub different parts of the application depending
on the features you are testing, GNATbdd might need to generate multiple
executables that will be run for the tests.
All of these will be examined in more details in this section.
The general workflow is thus the following::
GNATbdd library Ada step definitions
\ /
\ /
\ /
GNATbdd ------------ Application code
|
| generate glue code
| <compile and link>
|
\ feature files
\ /
\ /
\ /
test driver
Given its dependencies, GNATbdd (and the compiling of the driver) only need
to be performed when your application's code changes, or the step definitions
change. This step is not needed when you only change the features files
themselves, or even if you add new features files.
Compiling steps
===============
@ -12,50 +69,63 @@ and what the expected result should be.
This is performed via the various steps defined in the scenario. We now need
to associate those steps (English sentences) into actual code.
This is done in one or more Ada files which are compiled and linked into the
GNATbdd framework to generate an executable. This executable is then run and
dynamically parses all the :file:`*.feature` file as we described in the
previous step::
.. index:: gnatbdd switches; -P
Ada BDD library Ada step definitions
\ /
\ /
\ /
GNATbdd
| generate glue code
| <compile and link>
|
\ feature files
\ /
\ /
\ /
test driver (exec)
GNATbdd has one mandatory parameter, :option:`-P project.gpr`, which points to
the project file you are using to build your application.
GNATbdd will parse that project to find all its source files, and look for
step definitions in all of the spec files.
.. index:: switches; --steps
.. index:: gnatbdd switches; --steps
The step definitions are found automatically by parsing the Ada package specs
found in the directory :file:`features/step_definitions` (by default, although
you can use the :option:`--steps=DIR` switch to point to another directory. All
the Ada files found in those directories (recursively) will be parsed for
subprograms that have the Step_Regexp aspect or pragma (see below).
It is possible to mix the step definitions with your standard application code,
but this is not a recommended approach and it is better to separate the two.
Additionally, you can add one or more directories that are not part of your
project, but that should also be parsed (recursively). For instance, by
specifying :option:`--steps=features/step_definitions` one or more type,
that directory and all its subdirectories will be searched for Ada spec
files that might contain step definitions.
When you launch GNATbdd, it will search for all the Ada files in those
directories, generate one Ada file, and will then link your Ada files, the
generated file, and the relevant parts of the GNATbdd library. This will
generate a single executable that is automatically spawned and then in
charge of processing all your :file:`.feature` files.
directories and generate one Ada file, named by default :file:`driver.adb`.
That file is generated in the object directory of the project you passed
in argument. That directory should therefore be writable (although it will
be created automatically if it does not exist yet).
.. note::
This compilation is done by generating an extending project file, so users
will also have to provide a project file that points to all their sources.
.. index:: gnatbdd switches; --driver
If you wish to use another name for the driver (and therefore for the
generated executable), you can use the :option:`--driver=NAME` switch
on the command line.
Building the driver
===================
Once the driver has been generated by GNATbdd, you now need to compile
it. GNATbdd has in fact also generated a project file, named by default
:file:`driver.gpr` (that name is also set through the :option:`--driver`
switch).
So all you have to do is hopefully::
> gprbuild -P obj/driver.gpr
where :file:`obj/` is the object directory of the application's project.
The generated project depends on three other projects:
your application's project
This is referenced through an absolute path, so should always be found.
:file:`gnatbdd.gpr`
This is a project installed along with GNATbdd. It should be in one of
the directories looked for by the compiler (which is automatic if you
installed GNATbdd in the same directory as the compiler), or in one of
the directories part of the `GPR_PROJECT_PATH` environment variable.
:file:`gnatcoll.gpr`
This is the project file for the GNAT Components Collection, which should
be available the same way that :file:`gnatbdd.gpr` is.
As a result, the above compilation step will only take time whenever you
add or change step definitions, but not when you only add :file:`.feature`
files that reuse existing features.
Add files for step definitions
==============================
@ -99,20 +169,33 @@ It is recommended that regular expressions always be surrounded with '^' and
'$', to indicate they should match the whole step definition, and not just part
of it.
.. index:: switches; --duplicates
Some steps include extra information, like a table or a multi-line string.
This information is not part of the regular expression, although the
subprogram should have one or more parameters for it. For instance::
The switch :option:`--duplicates` can be used to systematically check all
regular expressions for each step, and warn when there are multiple matching
regexps.
with BDD.Tables; use BDD.Tables;
package My_Steps is
-- @then ^I should see the following results:$
procedure Check_Results (Expected : BDD.Tables.Table);
end My_Steps;
Here, GNATbdd will notice that the subprogram has one more parameter than
there are parenthesis groups in the regular expression. It then checks for
this extra parameter whether the type is `BDD.Tables.Table`. If this is the
case, that parameter will be passed the table that the user wrote as part
of the step.
The comparison of the type is purely textual, there is no semantic analysis.
So it might be specified exactly as `BDD.Tables.Table`, even if you are using
use clauses.
.. note::
alternatively, we could output the name/location of the subprogram that
handled the step, to ensure the right one is executed.
Assert library
==============
The intent is that the steps should raise an exception `Assert_Failure`
The intent is that the steps should raise an exception
when the step fails. GNATbdd provides the package :file:`BDD.Asserts` to help
perform the tests and raise the exception when they fail. This package will
also make sure a proper error message is logged, showing the expecting and
@ -139,10 +222,6 @@ Many more variants of `Assert` exist, which are able to compare a lot of
the usual Ada types, as well as more advanced types like lists of strings, or
the tables that are used in the feature files to provide data to steps.
.. note::
when comparing steps, the output should highlight the parts of the string
that are different, to help spot the difference, We also need to do
something special for trailing spaces.
Predefined Regular Expressions
==============================
@ -152,8 +231,8 @@ regular expressions that can be used in your own regular expressions. These
expressions have a name, that can be used in your regexps by using a leading
colon, as in::
procedure My_Step (Expected : Integer)
with Step_Regexp => "^I should get :natural results";
-- @then "^I should get :natural results";
procedure My_Step (Expected : Integer);
Here is the full list of predefined regular expressions:
@ -205,22 +284,6 @@ display possible definitions for the corresponding subprograms, which you can
copy and paste into your Ada file directly. This helps getting started.
Step timeout
============
.. note::
This will likely require each scenario to be run in its own task. There
will be only one such task, so it doesn't really add constraints on the
user code or step definitions, but it makes debugging slightly more
difficult.
.. index:: switches; --timeout
You can use the :option:`--timeout` switch to specify a maximum time that
steps can take to execute. A test that times out will automatically fail
with an appropriate error message.
Writing steps in python
=======================
@ -253,21 +316,6 @@ of the test. In particular, this is often necessary when testing graphical
user interfaces and other event-based applications.
Running tests in parallel
=========================
There are multiple modes to run features in parallel. The parallelism is always
between :file:`.feature` file, never between the Scenario of a given file.
When your application is task safe, you can run multiple features in parallel
by running each in its own task (up to a maximum number of tasks of course).
In other cases, GNATbdd can automatically spawn several instances of the test
driver, each running a single feature in parallel of the others. This is
slightly less efficient, but does not impose task-safety constraints on
your application.
White box vs Black box testing
==============================
@ -286,10 +334,10 @@ The main advantage is that the application is tested exactly as the user would
use it. This mode is compatible with most applications, like command-line
application, graphical user interfaces, web servers or embedded applications.
When testing embedded applications, GNATbdd will run on the host, and the
application will be spawned on the target. Communication between the two is the
responsibility of the step definition, and could take the form of examining the
standard output or communicating via sockets for instance.
When testing embedded applications, the test driver will run on the host, and
the application will be spawned on the target. Communication between the two is
the responsibility of the step definition, and could take the form of examining
the standard output or communicating via sockets for instance.
No real restriction apply to the way the step definition is written, since it
is running on the host, not in the more limited environment of the target.
@ -307,7 +355,7 @@ dragging in a lot of the application's code, which makes the link time for
the driver longer.
More importantly, this mode might not be compatible with embedded development,
since GNATbdd runs on the host.
since the driver runs on the host.
.. note::
Can we run the steps directly on the target in this case, while limiting

View File

@ -119,7 +119,7 @@ package body BDD is
Define_Switch
(Config,
Switch => "-o=",
Switch => "-o:",
Argument => "file",
Help => "Where the output should be sent (stdout by default)."
& " If the extension is .html, the output is HTML.");