mirror of https://github.com/briot/gnatbdd
465 lines
18 KiB
ReStructuredText
465 lines
18 KiB
ReStructuredText
*****************
|
|
Steps definitions
|
|
*****************
|
|
|
|
GNATbdd
|
|
=======
|
|
|
|
GNATbdd is the first tool that you will need to run as part of your testing
|
|
framework.
|
|
|
|
Its role is to aggregate, into one or more executables various pieces of
|
|
code:
|
|
|
|
Part of your application's code
|
|
This is the code you intend to test, and all its closure and dependencies.
|
|
|
|
Step definitions
|
|
These are (hopefully short) subprograms that create the link between the
|
|
English sentences you wrote in your features files, and the actual code to
|
|
execute.
|
|
|
|
GNATbdd's own library
|
|
GNATbdd comes with an extensive library which includes looking for features
|
|
files to run, parsing them, actually running them (or a subset of them
|
|
depending on command line switches), and an assert library to help you
|
|
write your own tests.
|
|
|
|
Stubs
|
|
It is often useful, when running test, to stub a part of your application.
|
|
This might be done to simplify the test setup (no need for a remove server
|
|
for instance for a distributed application), to speed up the test by
|
|
abstracting a slower part of the application, or to help focus the tests.
|
|
Since you might want to stub different parts of the application depending
|
|
on the features you are testing, GNATbdd might need to generate multiple
|
|
executables that will be run for the tests.
|
|
|
|
All of these will be examined in more details in this section.
|
|
|
|
The general workflow is thus the following::
|
|
|
|
GNATbdd library Ada step definitions
|
|
\ /
|
|
\ /
|
|
\ /
|
|
GNATbdd ------------ Application code
|
|
|
|
|
| generate glue code
|
|
| <compile and link>
|
|
|
|
|
\ feature files
|
|
\ /
|
|
\ /
|
|
\ /
|
|
test driver
|
|
|
|
|
|
Given its dependencies, GNATbdd (and the compiling of the driver) only need
|
|
to be performed when your application's code changes, or the step definitions
|
|
change. This step is not needed when you only change the features files
|
|
themselves, or even if you add new features files.
|
|
|
|
|
|
Compiling steps
|
|
===============
|
|
|
|
Now that you have described in the features file what the test should be doing
|
|
and what the expected result should be.
|
|
|
|
This is performed via the various steps defined in the scenario. We now need
|
|
to associate those steps (English sentences) into actual code.
|
|
|
|
.. index:: gnatbdd switches; -P
|
|
|
|
GNATbdd has one mandatory parameter, :option:`-P project.gpr`, which points to
|
|
the project file you are using to build your application.
|
|
|
|
GNATbdd will parse that project to find all its source files, and look for
|
|
step definitions in all of the spec files.
|
|
|
|
.. index:: gnatbdd switches; --steps
|
|
|
|
Additionally, you can add one or more directories that are not part of your
|
|
project, but that should also be parsed (recursively). For instance, by
|
|
specifying :option:`--steps=features/step_definitions` one or more type,
|
|
that directory and all its subdirectories will be searched for Ada spec
|
|
files that might contain step definitions.
|
|
|
|
When you launch GNATbdd, it will search for all the Ada files in those
|
|
directories and generate one Ada file, named by default :file:`driver.adb`.
|
|
That file is generated in the object directory of the project you passed
|
|
in argument. That directory should therefore be writable (although it will
|
|
be created automatically if it does not exist yet).
|
|
|
|
.. index:: gnatbdd switches; --driver
|
|
|
|
If you wish to use another name for the driver (and therefore for the
|
|
generated executable), you can use the :option:`--driver=NAME` switch
|
|
on the command line.
|
|
|
|
Building the driver
|
|
===================
|
|
|
|
Once the driver has been generated by GNATbdd, you now need to compile
|
|
it. GNATbdd has in fact also generated a project file, named by default
|
|
:file:`driver.gpr` (that name is also set through the :option:`--driver`
|
|
switch).
|
|
|
|
So all you have to do is hopefully::
|
|
|
|
> gprbuild -P obj/driver.gpr
|
|
|
|
where :file:`obj/` is the object directory of the application's project.
|
|
|
|
The generated project depends on three other projects:
|
|
|
|
your application's project
|
|
This is referenced through an absolute path, so should always be found.
|
|
|
|
:file:`gnatbdd.gpr`
|
|
This is a project installed along with GNATbdd. It should be in one of
|
|
the directories looked for by the compiler (which is automatic if you
|
|
installed GNATbdd in the same directory as the compiler), or in one of
|
|
the directories part of the `GPR_PROJECT_PATH` environment variable.
|
|
|
|
:file:`gnatcoll.gpr`
|
|
This is the project file for the GNAT Components Collection, which should
|
|
be available the same way that :file:`gnatbdd.gpr` is.
|
|
|
|
|
|
Add files for step definitions
|
|
==============================
|
|
|
|
The steps themselves can basically perform any action you want, and they
|
|
define whether you are doing black box or white box testing (see below).
|
|
|
|
.. highlight:: ada
|
|
|
|
The Ada packages should contain code similar to the following::
|
|
|
|
package My_Steps is
|
|
|
|
-- @given ^a user named (.*)$
|
|
procedure Set_User (Name : String);
|
|
|
|
-- @then ^I should get (\d+) as a result$
|
|
procedure Check_Result (Expected : Integer);
|
|
end My_Steps;
|
|
|
|
The example above shows two steps defined with special comments.
|
|
The comment must occur just before the subprogram to which it applies.
|
|
|
|
.. note::
|
|
Should support custom aspects ?
|
|
With comments, how do we handle cases where the regexp is too long
|
|
to fix on a line, except for using pragma Style_Checks(Off).
|
|
|
|
The comments should start with one of '@given', '@then', '@and', '@but'
|
|
or '@when'.
|
|
There is no semantic difference, they only act as a way to help
|
|
introduce the regexp.
|
|
|
|
It is recommended that regular expressions always be surrounded with '^' and
|
|
'$', to indicate they should match the whole step definition, and not just part
|
|
of it.
|
|
|
|
Parameter types
|
|
===============
|
|
|
|
The regular expressions are matched with the step as found in the
|
|
:file:`*.feature` file. The parenthesis groups found in the regexp will be
|
|
passed as parameters to the procedure. By default, all parameters are passed as
|
|
strings. If you use another scalar type for the parameter, GNATbdd will use a
|
|
`Type'Value (...)` before passing the parameter, and raise an error if
|
|
the type is incorrect.
|
|
|
|
GNATbdd provides specific handling for a few parameter types. Note that the
|
|
type must be written exactly as in the table below (case-insensitive), since
|
|
GNATbdd does not contain a semantic analyzer to resolve names.
|
|
|
|
+-------------------+-----------------------------------------------------+
|
|
| Type | Conversion from regexp match |
|
|
+===================+=====================================================+
|
|
| String | The parenthesis group as matched by the regexp |
|
|
+-------------------+-----------------------------------------------------+
|
|
| Ada.Calendar.Time | Converts a date with GNATCOLL.Utils.Time_Value. |
|
|
| | This supports a number of date and time formats, for|
|
|
| | instance '2015-06-15T12:00:00Z' or '2015-06-15' or |
|
|
| | '15 jun 2015' or 'jun 15, 2015'. |
|
|
+-------------------+-----------------------------------------------------+
|
|
| others | Use others'Value to convert from string |
|
|
+-------------------+-----------------------------------------------------+
|
|
|
|
.. _using_tables_in_step_definitions:
|
|
|
|
Using tables in step definitions
|
|
--------------------------------
|
|
|
|
Some steps include extra information, like a table or a multi-line string.
|
|
This information is not part of the regular expression, although the
|
|
subprogram should have one or more parameters for it. For instance::
|
|
|
|
with BDD.Tables; use BDD.Tables;
|
|
package My_Steps is
|
|
|
|
-- @then ^I should see the following results:$
|
|
procedure Check_Results (Expected : BDD.Tables.Table);
|
|
|
|
end My_Steps;
|
|
|
|
Here, GNATbdd will notice that the subprogram has one more parameter than
|
|
there are parenthesis groups in the regular expression. It then checks for
|
|
this extra parameter whether the type is `BDD.Tables.Table`. If this is the
|
|
case, that parameter will be passed the table that the user wrote as part
|
|
of the step.
|
|
|
|
The comparison of the type is purely textual, there is no semantic analysis.
|
|
So it might be specified exactly as `BDD.Tables.Table`, even if you are using
|
|
use clauses.
|
|
|
|
|
|
Assert library
|
|
==============
|
|
|
|
The intent is that the steps should raise an exception
|
|
when the step fails. GNATbdd provides the package :file:`BDD.Asserts` to help
|
|
perform the tests and raise the exception when they fail. This package will
|
|
also make sure a proper error message is logged, showing the expecting and
|
|
actual outputs.
|
|
|
|
For instance, the implementation for one of the steps above could be::
|
|
|
|
with BDD.Asserts; use BDD.Asserts;
|
|
|
|
package body My_Steps is
|
|
procedure Check_Result (Expected : Integer) is
|
|
Actual : constant Integer := Get_Current_Result;
|
|
begin
|
|
Assert (Expected, Actual, "Incorrect result");
|
|
end Check_Result;
|
|
end My_Steps;
|
|
|
|
When this test fails, GNATbdd will display extra information, as in::
|
|
|
|
Then I should get 5 as result # [FAILED]
|
|
Incorrect result: 5 /= 4 at my_steps.adb:7
|
|
|
|
Many more variants of `Assert` exist, which are able to compare a lot of
|
|
the usual Ada types, as well as more advanced types like lists of strings, or
|
|
the tables that are used in the feature files to provide data to steps.
|
|
|
|
Automatic type conversion
|
|
=========================
|
|
|
|
By default, all the parenthesis group in your regular expressions are
|
|
associated with `String` parameters in the subprogram that implements the
|
|
step.
|
|
|
|
However, GNATbdd also accepts other types for parameters, and will
|
|
automatically convert the string to them. The types are matched with string
|
|
comparison, so they must be defined exactly as how they appear in the following
|
|
table (casing not withstanding), even if you are using use clauses in your
|
|
packages.
|
|
|
|
+-------------------+-----------------------------------------------------+
|
|
| Type | Description |
|
|
+===================+=====================================================+
|
|
| String | The default parameter type |
|
|
+-------------------+-----------------------------------------------------+
|
|
| Integer | Typically associated with (\d+) in the regexp |
|
|
| Natural | |
|
|
+-------------------+-----------------------------------------------------+
|
|
| Ada.Calendar.Time | Date (including optional time and timezone) |
|
|
| | "2014-01-01 13:00:00+01:00" |
|
|
| | "Thu, 19 Dec 2014 13:59:12" |
|
|
| | "19/12/2014" |
|
|
| | "12/19/2014" |
|
|
+-------------------+-----------------------------------------------------+
|
|
| BDD.Tables.Table | See :ref:`using_tables_in_step_definitions` |
|
|
+-------------------+-----------------------------------------------------+
|
|
| other types | GNATbdd will generate a type'Image call |
|
|
+-------------------+-----------------------------------------------------+
|
|
|
|
|
|
Predefined Regular Expressions
|
|
==============================
|
|
|
|
To simplify the writting of your steps, GNATbdd provides a number of predefined
|
|
regular expressions that can be used in your own regular expressions. These
|
|
expressions have a name, that can be used in your regexps by using a leading
|
|
percent sign, as in::
|
|
|
|
-- @then "^I should get %natural results";
|
|
procedure My_Step (Expected : Integer);
|
|
|
|
The predefined regexps are automatically included in a parenthesis group,
|
|
so you should not add parenthesis yourself.
|
|
|
|
Here is the full list of predefined regular expressions:
|
|
|
|
+---------+----------------------------+-----------------------------+
|
|
| name | examples | Ada type |
|
|
+=========+============================+=============================+
|
|
| integer | -1; 0; 234 | String or Integer |
|
|
+---------+----------------------------+-----------------------------+
|
|
| float | -1.0E+10; 002E-10 | String or Float |
|
|
+---------+----------------------------+-----------------------------+
|
|
| natural | 2; 56 | String or Natural |
|
|
+---------+----------------------------+-----------------------------+
|
|
| date | Feb 04, 2014; 2014-02-04 | String or Ada.Calendar.Time |
|
|
+---------+----------------------------+-----------------------------+
|
|
|
|
You can use the percent symbol twice ("%%") when you need to ignore a
|
|
string matching one of the predefined regular expressions. For instance,
|
|
"%%integer" would match the string "%integer" in the feature file. When
|
|
you only when to insert a percent sign before any other string, you do not
|
|
need to duplicate it though (so "%test" can be written exactly as is).
|
|
|
|
Predefined steps
|
|
================
|
|
|
|
GNATbdd itself includes some predefined steps, which you can immediately use
|
|
in your :file:`.feature` file.
|
|
|
|
Here is the full list of predefined steps:
|
|
|
|
* `When I run '.*'`
|
|
|
|
This step can be used to spawn an executable (possibly with its arguments) on
|
|
the local machine.
|
|
|
|
* `Then (stdout|stderr) should be .*`
|
|
|
|
Compare the contents of standard output or standard error with an expected
|
|
output. In general, the last argument would be specified as a multi-string
|
|
argument in your :file:`.feature` file.
|
|
|
|
|
|
.. note::
|
|
When an application is using AWS, we could have predefined steps to connect
|
|
to a web server and compare its output (json, html,...)
|
|
|
|
|
|
Missing step definitions
|
|
========================
|
|
|
|
When the :file:`*.feature` files contain steps that have no corresponding
|
|
definition, they are are highlighted in a special color, and GNATbdd will
|
|
display possible definitions for the corresponding subprograms, which you can
|
|
copy and paste into your Ada file directly. This helps getting started.
|
|
|
|
|
|
Writing steps in python
|
|
=======================
|
|
|
|
.. highlight:: python
|
|
|
|
Steps can also be defined in python, by creating python files with contents
|
|
similar to::
|
|
|
|
@step_regexp("^I should get :num as a result")
|
|
def check_result(expected):
|
|
current = get_current_result()
|
|
if expected != current:
|
|
raise AssertionError(
|
|
"Invalid result %d != %d" % (expected, current))
|
|
|
|
As usual, any python file found in the :file:`features/step_definitions`
|
|
directory or the one set through :option:`--steps` will be analyzed,
|
|
and those that use the `@step_regexp` decorator.
|
|
|
|
Set up
|
|
------
|
|
|
|
Often, the code for the step definitions will need some initialization
|
|
(connecting to a database, opening a web browser,...). It is possible
|
|
to declare any number of such initialization subprograms.
|
|
|
|
Some of them will be run once per execution of the driver, others will
|
|
be executed one per feature file.
|
|
|
|
These subprograms are also declared in the step definitions files, as
|
|
in::
|
|
|
|
package My_Steps is
|
|
|
|
@setup_driver
|
|
procedure Init_Driver;
|
|
|
|
@setup_feature
|
|
procedure Init_Feature;
|
|
|
|
end My_Steps;
|
|
|
|
The subprograms never take any parameter. Since they might be called
|
|
multiple times, they also need to clean up any initialization they might
|
|
have done before.
|
|
|
|
Most of the time, it is better for these procedures to be written directly
|
|
as specific steps in the tests, so `@setup_feature` should in general be
|
|
replaced with a `Background` section in the feature file.
|
|
|
|
Asynchronous tests
|
|
==================
|
|
|
|
.. note::
|
|
Note sure how to implement this yet.
|
|
|
|
In some cases, it is necessary to stop executing steps to give some time for
|
|
the application to complete its handling, and then come back to the execution
|
|
of the test. In particular, this is often necessary when testing graphical
|
|
user interfaces and other event-based applications.
|
|
|
|
|
|
White box vs Black box testing
|
|
==============================
|
|
|
|
Testing can be done in various ways, and this section tries to provide a
|
|
few leads on how to organize and perform your tests.
|
|
|
|
Black box testing
|
|
-----------------
|
|
|
|
In this mode, the application is spawned with specific arguments, and all
|
|
interaction with it (input or output) is done only as a user would. It is not
|
|
possible to examine the value of specific variables, unless they have a direct
|
|
impact on what can be seen from the outside.
|
|
|
|
The main advantage is that the application is tested exactly as the user would
|
|
use it. This mode is compatible with most applications, like command-line
|
|
application, graphical user interfaces, web servers or embedded applications.
|
|
|
|
When testing embedded applications, the test driver will run on the host, and
|
|
the application will be spawned on the target. Communication between the two is
|
|
the responsibility of the step definition, and could take the form of examining
|
|
the standard output or communicating via sockets for instance.
|
|
|
|
No real restriction apply to the way the step definition is written, since it
|
|
is running on the host, not in the more limited environment of the target.
|
|
|
|
White box testing
|
|
-----------------
|
|
|
|
In this mode, the step definitions can access all the public parts of your
|
|
application's code (or at least the public part of it). As a result, it is
|
|
possible to inspect in details the actual start of your application, and
|
|
perhaps catch errors earlier in the code.
|
|
|
|
One of the inconvenients in this mode is that the steps themselves end up
|
|
dragging in a lot of the application's code, which makes the link time for
|
|
the driver longer.
|
|
|
|
More importantly, this mode might not be compatible with embedded development,
|
|
since the driver runs on the host.
|
|
|
|
.. note::
|
|
Can we run the steps directly on the target in this case, while limiting
|
|
what features of the code we use like controlled types, memory
|
|
allocation,...
|
|
|
|
White box testing can itself be done in one of two ways: either be linking
|
|
the application's code within the GNATbdd driver (because the code for the
|
|
steps would `with` the application's own packages), or by spawning an
|
|
executable and communicating with it via various means (stdin/stdout,
|
|
sockets, pipes,...)
|