From dfc490af5a06157d79a2dd41e07a5459247117fa Mon Sep 17 00:00:00 2001 From: Emmanuel Briot Date: Fri, 6 Jun 2014 11:40:41 +0200 Subject: [PATCH] Clarify the role of gnatbdd and the driver in the documentation --- docs/driver.rst | 87 ++++++++++++++++++ docs/features.rst | 45 +--------- docs/index.rst | 1 + docs/output.rst | 54 ++++++------ docs/steps.rst | 220 ++++++++++++++++++++++++++++------------------ src/bdd/bdd.adb | 2 +- 6 files changed, 251 insertions(+), 158 deletions(-) create mode 100644 docs/driver.rst diff --git a/docs/driver.rst b/docs/driver.rst new file mode 100644 index 0000000..1605cbe --- /dev/null +++ b/docs/driver.rst @@ -0,0 +1,87 @@ +*************** +The test driver +*************** + +The test driver is the executable generated by GNATbdd. In fact, there could +be several drivers built, depending on how you chose to run GNATbdd. For +instance, one of the driver might stub part of your application, whereas +another driver would test that same part and stub another part. + +Finding features +================ + +Let's first examine how the driver finds all the features for your application. + +.. index:: switches; --ext + +Typically, the features are written in text files with a :file:`.feature` +extension. Several editors will automatically provide syntax highlighting with +such an extension. However, you can choose your own extension by using the +:option:`--ext .EXT` switch when you run the driver. + +.. index:: switches; --features + +By default, the driver will find all the files with a matching extension in +a subdirectory names :file:`features`, within the directory where the driver +is run. You can however specify one or more other directories to check, +by using the :option:`--features DIR` switch one or more times. + +The driver will always search recursively in those directories. This lets you +organize the files into subdirectories when you have several of them, and +thus perhaps make it easier to refer to them. + +The driver will run the features file in alphabetical order of the full path of +the feature file, by default. This provides a more consistent output when +the driver is run multiple times. + +.. index:: switches; --random + +However, by using the :option:`--random` switch, the order of features is +randomized. This is a convenient way to check that the various tests are indeed +independent of each other, as they should be to make it possible to run tests +individually. + +Running feature +=============== + +.. index:: switches; --tags + +When you run the driver, you can use the switch :option:`--tags` to control +which scenarios should be run. For instance, all scenarios related to '@gui', +or all scenarios not related to '@startup'. You can of course select subsets of +scenarios based on the file names, but tags provide a file-system-agnostic +selection mechanism. + +Step timeout +============ + +.. note:: + This will likely require each scenario to be run in its own task. There + will be only one such task, so it doesn't really add constraints on the + user code or step definitions, but it makes debugging slightly more + difficult. + +.. note:: + Perhaps we can also change the way the driver is generated, rather than do + everything dynamically in the driver itself. + +.. index:: switches; --timeout + +You can use the :option:`--timeout` switch to specify a maximum time that +steps can take to execute. A test that times out will automatically fail +with an appropriate error message. + + +Running tests in parallel +========================= + +There are multiple modes to run features in parallel. The parallelism is always +between :file:`.feature` file, never between the Scenario of a given file. + +When your application is task safe, you can run multiple features in parallel +by running each in its own task (up to a maximum number of tasks of course). + +In other cases, the driver can automatically spawn several instances of itself, +each running a single feature in parallel of the others. This is slightly less +efficient, but does not impose task-safety constraints on your application. + diff --git a/docs/features.rst b/docs/features.rst index eb4c025..3169a56 100644 --- a/docs/features.rst +++ b/docs/features.rst @@ -12,40 +12,6 @@ meant for human readers, and is not used by the automatic tool. The features are then further divided into one or more scenarios, which are sometimes also called "tests". These will be discussed in the next section. -Finding features -================ - -Let's first examine how GNATbdd finds all the features for your application. - -.. index:: switches; --ext - -Typically, the features are written in text files with a :file:`.feature` -extension. Several editors will automatically provide syntax highlighting with -such an extension. However, you can choose your own extension by using the -:option:`--ext .EXT` switch when you run GNATbdd. - -.. index:: switches; --features - -By default, GNATbdd will find all the files with a matching extension in -a subdirectory names :file:`features`, within the directory where GNATbdd -is run. You can however specify one or more other directories to check, -by using the :option:`--features DIR` switch one or more times. - -GNATbdd will always search recursively in those directories. This lets you -organize the files into subdirectories when you have several of them, and -thus perhaps make it easier to refer to them. - -GNATbdd will run the features file in alphabetical order of the full path of -the feature file, by default. This provides a more consistent output when -GNATbdd is run multiple times. - -.. index:: switches; --random - -However, by using the :option:`--random` switch, the order of features is -randomized. This is a convenient way to check that the various tests are indeed -independent of each other, as they should be to make it possible to run tests -individually. - Syntax of the features file =========================== @@ -123,20 +89,11 @@ to run subsets of the whole set of scenarios. Here is an example:: The tags of the feature automatically apply to its Scenarios -.. index:: switches; --tags - -When you run GNATbdd, you can use the switch :option:`--tags` to control which -scenarios should be run. For instance, all scenarios related to '@gui', or all -scenarios not related to '@startup'. You can of course select subsets of -scenarios based on the file names, but tags provide a file-system-agnostic -selection mechanism. - - Other usage of tags could be to identify *slow tests* (with @slow) so that their timeout is increased. A tag can also be used to link a scenario to a *high-level requirement* in your -application +application, for instance @HLR-12-2. Tags can also be used to identify *expected failures* (for instance @xfail), or *work in progress* (for instance @wip). diff --git a/docs/index.rst b/docs/index.rst index 6ffee48..958ef94 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -13,6 +13,7 @@ GNAT Behavior Driven Development bdd features steps + driver output customizing ideas diff --git a/docs/output.rst b/docs/output.rst index 0eea172..b620a22 100644 --- a/docs/output.rst +++ b/docs/output.rst @@ -2,7 +2,7 @@ Controlling the output ********************** -GNATbdd provides a rich set of options for controlling the output. +The driver provides a rich set of options for controlling the output. Colors ====== @@ -10,7 +10,7 @@ Colors .. index:: switches; --color Color output is enabled by default when the terminal supports it. It is used to -highlight passing, failing and skipped tests. GNATbdd attempts to detect +highlight passing, failing and skipped tests. The driver attempts to detect automatically whether advanced features like colors or moving the cursor are supported. However, you can also force a specific setting by using one of: @@ -18,14 +18,14 @@ supported. However, you can also force a specific setting by using one of: Forces the use of colors in the output. This is in particular useful when the output is redirected to a file which you intend to display in a console later - on. GNATbdd always uses ANSI escape sequences when redirecting to a file, + on. The driver always uses ANSI escape sequences when redirecting to a file, so on Windows you should install `ANSICON `_. This is not necessary when you are directly displaying the output in the console though. * :option:`--color=no` - Disable the use of colors. GNATbdd will output strings such as "[OK]", + Disable the use of colors. The driver will output strings such as "[OK]", "[FAILED]" or "[SKIPPED]" instead. * :option:`--color=auto` (the default) @@ -45,7 +45,7 @@ as possible directly in the console, so that hopefully it is easy to undertand why a test is failing. In fact, some users might want to see the passing tests as well, just to make sure the testsuite is moving forward. -However, during an automatic test, all the output can be logged, and we are +However, during an automatic test, all the output can be dropped, and we are only interested in getting a final count of passing and failing tests. All these behavior can be controlled with the following command line @@ -53,7 +53,7 @@ switches: * :option:`--output=quiet` - GNATbdd only displays the final count for each categories of + The driver only displays the final count for each categories of tests (passed, failed and skipped). The output therefore looks like:: @@ -62,11 +62,11 @@ switches: * :option:`--output=dots` (the default) - GNATbdd displays a "." for each test that passes, a "F" for each + The driver displays a "." for each test that passes, a "F" for each test that fails (and will then display a full backtrace for those) and a "S" for skipped tests. - When all features have run, GNATbdd displays details for all tests + When all features have run, the driver displays details for all tests that did not pass. Finally, it displays a summary for the final count for each category. @@ -87,26 +87,40 @@ switches: * :option:`--output=hide_passed` - In this mode, GNATbdd does not display anything when a scenario passes, + In this mode, the driver does not display anything when a scenario passes, but displays the same backtrace as above for failed or skipped. One advantage is that the output of the full traces is done after each scenario has been run, not as a summary after all of them have run. * :option:`--output=full` - In this mode, GNATbdd displays output of all scenarios, whether they + In this mode, the driver displays output of all scenarios, whether they pass or fail. The output looks like the above, but instead of displaying - a ".", GNATbdd outputs the full lists of steps for that scenario. + a ".", the driver outputs the full lists of steps for that scenario. - In this mode, GNATbdd outputs each step before it is being run, thus this + In this mode, the driver outputs each step before it is being run, thus this allows you to monitor which step is taking long to execute. +Redirecting the output +====================== + +.. index:: switches; -o + +By default, the driver will print its output on the screen. It is possible +though to print to a file by using the :option:`-o FILE` switch. + +.. index:: HTML output + +If the file ends up with the :file:`.html` extension, the driver will +generate HTML output rather than textual output. This will display nicer +in a web browser. + Log files ========= .. index:: switches; --log -In addition, GNATbdd can still generate log files which contain output +In addition, the driver can still generate log files which contain output similar to the above, as well as any output done by your application on stdout or stderr. @@ -120,20 +134,6 @@ subdirectory of the directory where GNATbdd is run. Should this be in the root project's object_dir, when GNATbdd is run with a project in argument ? - -Projects -======== - -.. index:: switches; -P - -GNATbdd needs to compile the step definitions your application provides. The -features files themselves are parsed dynamically and do not need compile. - -For the compilation to succeed, GNATbdd needs to find where the sources of the -steps are (see the :option:`--steps` switch), as well as all the sources on -which they depend. Such sources are found via a GNAT Project file, which can -be specified with the :option:`-P PROJECT.gpr` switch. - Exit status =========== diff --git a/docs/steps.rst b/docs/steps.rst index 9a89ff5..221cf9c 100644 --- a/docs/steps.rst +++ b/docs/steps.rst @@ -2,6 +2,63 @@ Steps definitions ***************** +GNATbdd +======= + +GNATbdd is the first tool that you will need to run as part of your testing +framework. + +Its role is to aggregate, into one or more executables various pieces of +code: + +Part of your application's code + This is the code you intend to test, and all its closure and dependencies. + +Step definitions + These are (hopefully short) subprograms that create the link between the + English sentences you wrote in your features files, and the actual code to + execute. + +GNATbdd's own library + GNATbdd comes with an extensive library which includes looking for features + files to run, parsing them, actually running them (or a subset of them + depending on command line switches), and an assert library to help you + write your own tests. + +Stubs + It is often useful, when running test, to stub a part of your application. + This might be done to simplify the test setup (no need for a remove server + for instance for a distributed application), to speed up the test by + abstracting a slower part of the application, or to help focus the tests. + Since you might want to stub different parts of the application depending + on the features you are testing, GNATbdd might need to generate multiple + executables that will be run for the tests. + +All of these will be examined in more details in this section. + +The general workflow is thus the following:: + + GNATbdd library Ada step definitions + \ / + \ / + \ / + GNATbdd ------------ Application code + | + | generate glue code + | + | + \ feature files + \ / + \ / + \ / + test driver + + +Given its dependencies, GNATbdd (and the compiling of the driver) only need +to be performed when your application's code changes, or the step definitions +change. This step is not needed when you only change the features files +themselves, or even if you add new features files. + Compiling steps =============== @@ -12,50 +69,63 @@ and what the expected result should be. This is performed via the various steps defined in the scenario. We now need to associate those steps (English sentences) into actual code. -This is done in one or more Ada files which are compiled and linked into the -GNATbdd framework to generate an executable. This executable is then run and -dynamically parses all the :file:`*.feature` file as we described in the -previous step:: +.. index:: gnatbdd switches; -P - Ada BDD library Ada step definitions - \ / - \ / - \ / - GNATbdd - | generate glue code - | - | - \ feature files - \ / - \ / - \ / - test driver (exec) +GNATbdd has one mandatory parameter, :option:`-P project.gpr`, which points to +the project file you are using to build your application. +GNATbdd will parse that project to find all its source files, and look for +step definitions in all of the spec files. -.. index:: switches; --steps +.. index:: gnatbdd switches; --steps -The step definitions are found automatically by parsing the Ada package specs -found in the directory :file:`features/step_definitions` (by default, although -you can use the :option:`--steps=DIR` switch to point to another directory. All -the Ada files found in those directories (recursively) will be parsed for -subprograms that have the Step_Regexp aspect or pragma (see below). - -It is possible to mix the step definitions with your standard application code, -but this is not a recommended approach and it is better to separate the two. +Additionally, you can add one or more directories that are not part of your +project, but that should also be parsed (recursively). For instance, by +specifying :option:`--steps=features/step_definitions` one or more type, +that directory and all its subdirectories will be searched for Ada spec +files that might contain step definitions. When you launch GNATbdd, it will search for all the Ada files in those -directories, generate one Ada file, and will then link your Ada files, the -generated file, and the relevant parts of the GNATbdd library. This will -generate a single executable that is automatically spawned and then in -charge of processing all your :file:`.feature` files. +directories and generate one Ada file, named by default :file:`driver.adb`. +That file is generated in the object directory of the project you passed +in argument. That directory should therefore be writable (although it will +be created automatically if it does not exist yet). -.. note:: - This compilation is done by generating an extending project file, so users - will also have to provide a project file that points to all their sources. +.. index:: gnatbdd switches; --driver + +If you wish to use another name for the driver (and therefore for the +generated executable), you can use the :option:`--driver=NAME` switch +on the command line. + +Building the driver +=================== + +Once the driver has been generated by GNATbdd, you now need to compile +it. GNATbdd has in fact also generated a project file, named by default +:file:`driver.gpr` (that name is also set through the :option:`--driver` +switch). + +So all you have to do is hopefully:: + + > gprbuild -P obj/driver.gpr + +where :file:`obj/` is the object directory of the application's project. + +The generated project depends on three other projects: + +your application's project + This is referenced through an absolute path, so should always be found. + +:file:`gnatbdd.gpr` + This is a project installed along with GNATbdd. It should be in one of + the directories looked for by the compiler (which is automatic if you + installed GNATbdd in the same directory as the compiler), or in one of + the directories part of the `GPR_PROJECT_PATH` environment variable. + +:file:`gnatcoll.gpr` + This is the project file for the GNAT Components Collection, which should + be available the same way that :file:`gnatbdd.gpr` is. -As a result, the above compilation step will only take time whenever you -add or change step definitions, but not when you only add :file:`.feature` -files that reuse existing features. Add files for step definitions ============================== @@ -99,20 +169,33 @@ It is recommended that regular expressions always be surrounded with '^' and '$', to indicate they should match the whole step definition, and not just part of it. -.. index:: switches; --duplicates +Some steps include extra information, like a table or a multi-line string. +This information is not part of the regular expression, although the +subprogram should have one or more parameters for it. For instance:: -The switch :option:`--duplicates` can be used to systematically check all -regular expressions for each step, and warn when there are multiple matching -regexps. + with BDD.Tables; use BDD.Tables; + package My_Steps is + + -- @then ^I should see the following results:$ + procedure Check_Results (Expected : BDD.Tables.Table); + + end My_Steps; + +Here, GNATbdd will notice that the subprogram has one more parameter than +there are parenthesis groups in the regular expression. It then checks for +this extra parameter whether the type is `BDD.Tables.Table`. If this is the +case, that parameter will be passed the table that the user wrote as part +of the step. + +The comparison of the type is purely textual, there is no semantic analysis. +So it might be specified exactly as `BDD.Tables.Table`, even if you are using +use clauses. -.. note:: - alternatively, we could output the name/location of the subprogram that - handled the step, to ensure the right one is executed. Assert library ============== -The intent is that the steps should raise an exception `Assert_Failure` +The intent is that the steps should raise an exception when the step fails. GNATbdd provides the package :file:`BDD.Asserts` to help perform the tests and raise the exception when they fail. This package will also make sure a proper error message is logged, showing the expecting and @@ -139,10 +222,6 @@ Many more variants of `Assert` exist, which are able to compare a lot of the usual Ada types, as well as more advanced types like lists of strings, or the tables that are used in the feature files to provide data to steps. -.. note:: - when comparing steps, the output should highlight the parts of the string - that are different, to help spot the difference, We also need to do - something special for trailing spaces. Predefined Regular Expressions ============================== @@ -152,8 +231,8 @@ regular expressions that can be used in your own regular expressions. These expressions have a name, that can be used in your regexps by using a leading colon, as in:: - procedure My_Step (Expected : Integer) - with Step_Regexp => "^I should get :natural results"; + -- @then "^I should get :natural results"; + procedure My_Step (Expected : Integer); Here is the full list of predefined regular expressions: @@ -205,22 +284,6 @@ display possible definitions for the corresponding subprograms, which you can copy and paste into your Ada file directly. This helps getting started. -Step timeout -============ - -.. note:: - This will likely require each scenario to be run in its own task. There - will be only one such task, so it doesn't really add constraints on the - user code or step definitions, but it makes debugging slightly more - difficult. - -.. index:: switches; --timeout - -You can use the :option:`--timeout` switch to specify a maximum time that -steps can take to execute. A test that times out will automatically fail -with an appropriate error message. - - Writing steps in python ======================= @@ -253,21 +316,6 @@ of the test. In particular, this is often necessary when testing graphical user interfaces and other event-based applications. -Running tests in parallel -========================= - -There are multiple modes to run features in parallel. The parallelism is always -between :file:`.feature` file, never between the Scenario of a given file. - -When your application is task safe, you can run multiple features in parallel -by running each in its own task (up to a maximum number of tasks of course). - -In other cases, GNATbdd can automatically spawn several instances of the test -driver, each running a single feature in parallel of the others. This is -slightly less efficient, but does not impose task-safety constraints on -your application. - - White box vs Black box testing ============================== @@ -286,10 +334,10 @@ The main advantage is that the application is tested exactly as the user would use it. This mode is compatible with most applications, like command-line application, graphical user interfaces, web servers or embedded applications. -When testing embedded applications, GNATbdd will run on the host, and the -application will be spawned on the target. Communication between the two is the -responsibility of the step definition, and could take the form of examining the -standard output or communicating via sockets for instance. +When testing embedded applications, the test driver will run on the host, and +the application will be spawned on the target. Communication between the two is +the responsibility of the step definition, and could take the form of examining +the standard output or communicating via sockets for instance. No real restriction apply to the way the step definition is written, since it is running on the host, not in the more limited environment of the target. @@ -307,7 +355,7 @@ dragging in a lot of the application's code, which makes the link time for the driver longer. More importantly, this mode might not be compatible with embedded development, -since GNATbdd runs on the host. +since the driver runs on the host. .. note:: Can we run the steps directly on the target in this case, while limiting diff --git a/src/bdd/bdd.adb b/src/bdd/bdd.adb index b20abc9..26f96bb 100644 --- a/src/bdd/bdd.adb +++ b/src/bdd/bdd.adb @@ -119,7 +119,7 @@ package body BDD is Define_Switch (Config, - Switch => "-o=", + Switch => "-o:", Argument => "file", Help => "Where the output should be sent (stdout by default)." & " If the extension is .html, the output is HTML.");