# Writing Tests

The following focuses on writing an integration test. However, writing unit
tests is also encouraged!

## Testsuite

Cargo has a wide variety of integration tests that execute the `cargo` binary
and verify its behavior, located in the [`testsuite`] directory.  The
[`support`] crate and [`snapbox`] contain many helpers to make this process easy.

There are two styles of tests that can roughly be categorized as
- functional tests
  - The fixture is programmatically defined
  - The assertions are regular string comparisons
  - Easier to share in an issue as a code block is completely self-contained
  - More resilient to insignificant changes though ui tests are easy to update when a change does occur
- ui tests
  - The fixture is file-based
  - The assertions use file-backed snapshots that can be updated with an env variable
  - Easier to review the expected behavior of the command as more details are included
  - Easier to get up and running from an existing project
  - Easier to reason about as everything is just files in the repo

These tests typically work by creating a temporary "project" with a
`Cargo.toml` file, executing the `cargo` binary process, and checking the
stdout and stderr output against the expected output.

### Functional Tests

Generally, a functional test will be placed in `tests/testsuite/<command>.rs` and will look roughly like:
```rust,ignore
#[cargo_test]
fn <description>() {
    let p = project()
        .file("src/main.rs", r#"fn main() { println!("hi!"); }"#)
        .build();

    p.cargo("run --bin foo")
        .with_stderr(
            "\
    [COMPILING] foo [..]
    [FINISHED] [..]
    [RUNNING] `target/debug/foo`
    ",
        )
        .with_stdout("hi!")
        .run();
    }
}
```

The [`#[cargo_test]` attribute](#cargo_test-attribute) is used in place of `#[test]` to inject some setup code.

[`ProjectBuilder`] via `project()`:
- Each project is in a separate directory in the sandbox
- If you do not specify a `Cargo.toml` manifest using `file()`, one is
  automatically created with a project name of `foo` using `basic_manifest()`.

[`Execs`] via `p.cargo(...)`:
- This executes the command and evaluates different assertions
  - See [`support::compare`] for an explanation of the string pattern matching.
    Patterns are used to make it easier to match against the expected output.

#### `#[cargo_test]` attribute

The `#[cargo_test]` attribute injects code which does some setup before starting the test.
It will create a filesystem "sandbox" under the "cargo integration test" directory for each test, such as `/path/to/cargo/target/tmp/cit/t123/`.
The sandbox will contain a `home` directory that will be used instead of your normal home directory.

The `#[cargo_test]` attribute takes several options that will affect how the test is generated.
They are listed in parentheses separated with commas, such as:

```rust,ignore
#[cargo_test(nightly, reason = "-Zfoo is unstable")]
```

The options it supports are:

* `nightly` — This will cause the test to be ignored if not running on the nightly toolchain.
  This is useful for tests that use unstable options in `rustc` or `rustdoc`.
  These tests are run in Cargo's CI, but are disabled in rust-lang/rust's CI due to the difficulty of updating both repos simultaneously.
  A `reason` field is required to explain why it is nightly-only.
* `build_std_real` — This is a "real" `-Zbuild-std` test (in the `build_std` integration test).
  This only runs on nightly, and only if the environment variable `CARGO_RUN_BUILD_STD_TESTS` is set (these tests on run on Linux).
* `build_std_mock` — This is a "mock" `-Zbuild-std` test (which uses a mock standard library).
  This only runs on nightly, and is disabled for windows-gnu.
* `requires_` — This indicates a command that is required to be installed to be run.
  For example, `requires_rustfmt` means the test will only run if the executable `rustfmt` is installed.
  These tests are *always* run on CI.
  This is mainly used to avoid requiring contributors from having every dependency installed.
* `>=1.64` — This indicates that the test will only run with the given version of `rustc` or newer.
  This can be used when a new `rustc` feature has been stabilized that the test depends on.
  If this is specified, a `reason` is required to explain why it is being checked.

#### Testing Nightly Features

If you are testing a Cargo feature that only works on "nightly" Cargo, then
you need to call `masquerade_as_nightly_cargo` on the process builder and pass 
the name of the feature as the reason, like this:

```rust,ignore
p.cargo("build").masquerade_as_nightly_cargo(&["print-im-a-teapot"])
```

If you are testing a feature that only works on *nightly rustc* (such as
benchmarks), then you should use the `nightly` option of the `cargo_test`
attribute, like this:

```rust,ignore
#[cargo_test(nightly, reason = "-Zfoo is unstable")]
```

This will cause the test to be ignored if not running on the nightly toolchain.

#### Specifying Dependencies

You should not write any tests that use the network such as contacting
crates.io. Typically, simple path dependencies are the easiest way to add a
dependency. Example:

```rust,ignore
let p = project()
    .file("Cargo.toml", r#"
        [package]
        name = "foo"
        version = "1.0.0"

        [dependencies]
        bar = {path = "bar"}
    "#)
    .file("src/lib.rs", "extern crate bar;")
    .file("bar/Cargo.toml", &basic_manifest("bar", "1.0.0"))
    .file("bar/src/lib.rs", "")
    .build();
```

If you need to test with registry dependencies, see
[`support::registry::Package`] for creating packages you can depend on.

If you need to test git dependencies, see [`support::git`] to create a git
dependency.

### UI Tests

UI Tests are a bit more spread out and generally look like:

`tests/testsuite/<command>/mod.rs`:
```rust,ignore
mod <case>;
```

`tests/testsuite/<command>/<case>/mod.rs`:
```rust,ignore
use cargo_test_support::prelude::*;
use cargo_test_support::compare::assert_ui;
use cargo_test_support::Project;
use cargo_test_support::curr_dir;

#[cargo_test]
fn <name>() {
    let project = Project::from_template(curr_dir!().join("in"));
    let project_root = project.root();
    let cwd = &project_root;

    snapbox::cmd::Command::cargo_ui()
        .arg("run")
        .arg_line("--bin foo")
        .current_dir(cwd)
        .assert()
        .success()
        .stdout_matches_path(curr_dir!().join("stdout.log"))
        .stderr_matches_path(curr_dir!().join("stderr.log"));

    assert_ui().subset_matches(curr_dir!().join("out"), &project_root);
}
```

Then populate
- `tests/testsuite/<command>/<case>/in` with the project's directory structure
- `tests/testsuite/<command>/<case>/out` with the files you want verified
- `tests/testsuite/<command>/<case>/stdout.log` with nothing
- `tests/testsuite/<command>/<case>/stderr.log` with nothing

`#[cargo_test]`:
- This is used in place of `#[test]`
- This attribute injects code which does some setup before starting the
  test, creating a filesystem "sandbox" under the "cargo integration test"
  directory for each test such as
  `/path/to/cargo/target/cit/t123/`
- The sandbox will contain a `home` directory that will be used instead of your normal home directory

`Project`:
- The project is copied from a directory in the repo
- Each project is in a separate directory in the sandbox

[`Command`] via `Command::cargo_ui()`:
- Set up and run a command.

[`OutputAssert`] via `Command::assert()`:
- Perform assertions on the result of the [`Command`]

[`Assert`] via `assert_ui()`:
- Verify the command modified the file system as expected

#### Updating Snapshots

The project, stdout, and stderr snapshots can be updated by running with the
`SNAPSHOTS=overwrite` environment variable, like:
```console
$ SNAPSHOTS=overwrite cargo test
```

Be sure to check the snapshots to make sure they make sense.

#### Testing Nightly Features

If you are testing a Cargo feature that only works on "nightly" Cargo, then
you need to call `masquerade_as_nightly_cargo` on the process builder and pass
the name of the feature as the reason, like this:

```rust,ignore
    snapbox::cmd::Command::cargo()
        .masquerade_as_nightly_cargo(&["print-im-a-teapot"])
```

If you are testing a feature that only works on *nightly rustc* (such as
benchmarks), then you should use the `nightly` option of the `cargo_test`
attribute, like this:

```rust,ignore
#[cargo_test(nightly, reason = "-Zfoo is unstable")]
```

This will cause the test to be ignored if not running on the nightly toolchain.

### Platform-specific Notes

When checking output, use `/` for paths even on Windows: the actual output
of `\` on Windows will be replaced with `/`.

Be careful when executing binaries on Windows. You should not rename, delete,
or overwrite a binary immediately after running it. Under some conditions
Windows will fail with errors like "directory not empty" or "failed to remove"
or "access is denied".

## Debugging tests

In some cases, you may need to dig into a test that is not working as you
expect, or you just generally want to experiment within the sandbox
environment. The general process is:

1. Build the sandbox for the test you want to investigate. For example:

   `cargo test --test testsuite -- features2::inactivate_targets`.
2. In another terminal, head into the sandbox directory to inspect the files and run `cargo` directly.
    1. The sandbox directories start with `t0` for the first test.

       `cd target/tmp/cit/t0`
    2. Set up the environment so that the sandbox configuration takes effect:

       `export CARGO_HOME=$(pwd)/home/.cargo`
    3. Most tests create a `foo` project, so head into that:

       `cd foo`
3. Run whatever cargo command you want. See [Running Cargo] for more details
   on running the correct `cargo` process. Some examples:

   * `/path/to/my/cargo/target/debug/cargo check`
   * Using a debugger like `lldb` or `gdb`:
        1. `lldb /path/to/my/cargo/target/debug/cargo`
        2. Set a breakpoint, for example: `b generate_root_units`
        3. Run with arguments: `r check`

[`testsuite`]: https://github.com/rust-lang/cargo/tree/master/tests/testsuite/
[`ProjectBuilder`]: https://github.com/rust-lang/cargo/blob/d847468768446168b596f721844193afaaf9d3f2/crates/cargo-test-support/src/lib.rs#L196-L202
[`Execs`]: https://github.com/rust-lang/cargo/blob/d847468768446168b596f721844193afaaf9d3f2/crates/cargo-test-support/src/lib.rs#L531-L550
[`support`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/lib.rs
[`support::compare`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/compare.rs
[`support::registry::Package`]: https://github.com/rust-lang/cargo/blob/d847468768446168b596f721844193afaaf9d3f2/crates/cargo-test-support/src/registry.rs#L311-L389
[`support::git`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/git.rs
[Running Cargo]: ../process/working-on-cargo.md#running-cargo
[`snapbox`]: https://docs.rs/snapbox/latest/snapbox/
[`Command`]: https://docs.rs/snapbox/latest/snapbox/cmd/struct.Command.html
[`OutputAssert`]: https://docs.rs/snapbox/latest/snapbox/cmd/struct.OutputAssert.html
[`Assert`]: https://docs.rs/snapbox/latest/snapbox/struct.Assert.html
