sembr tests/running.md
This commit is contained in:
Tshepang Mbambo
parent
2ae4b0bdad
commit
299a13b060
1 changed files with 76 additions and 75 deletions
|
|
@ -1,19 +1,19 @@
|
|||
# Running tests
|
||||
|
||||
You can run the entire test collection using `x`. But note that running the
|
||||
*entire* test collection is almost never what you want to do during local
|
||||
development because it takes a really long time. For local development, see the
|
||||
subsection after on how to run a subset of tests.
|
||||
You can run the entire test collection using `x`.
|
||||
But note that running the *entire* test collection is almost never what you want to do during local
|
||||
development because it takes a really long time.
|
||||
For local development, see the subsection after on how to run a subset of tests.
|
||||
|
||||
<div class="warning">
|
||||
|
||||
Running plain `./x test` will build the stage 1 compiler and then run the whole
|
||||
test suite. This not only includes `tests/`, but also `library/`, `compiler/`,
|
||||
Running plain `./x test` will build the stage 1 compiler and then run the whole test suite.
|
||||
This not only includes `tests/`, but also `library/`, `compiler/`,
|
||||
`src/tools/` package tests and more.
|
||||
|
||||
You usually only want to run a subset of the test suites (or even a smaller set
|
||||
of tests than that) which you expect will exercise your changes. PR CI exercises
|
||||
a subset of test collections, and merge queue CI will exercise all of the test
|
||||
of tests than that) which you expect will exercise your changes.
|
||||
PR CI exercises a subset of test collections, and merge queue CI will exercise all of the test
|
||||
collection.
|
||||
|
||||
</div>
|
||||
|
|
@ -22,8 +22,8 @@ collection.
|
|||
./x test
|
||||
```
|
||||
|
||||
The test results are cached and previously successful tests are `ignored` during
|
||||
testing. The stdout/stderr contents as well as a timestamp file for every test
|
||||
The test results are cached and previously successful tests are `ignored` during testing.
|
||||
The stdout/stderr contents as well as a timestamp file for every test
|
||||
can be found under `build/<target-tuple>/test/` for the given
|
||||
`<target-tuple>`. To force-rerun a test (e.g. in case the test runner fails to
|
||||
notice a change) you can use the `--force-rerun` CLI option.
|
||||
|
|
@ -40,8 +40,8 @@ notice a change) you can use the `--force-rerun` CLI option.
|
|||
|
||||
## Running a subset of the test suites
|
||||
|
||||
When working on a specific PR, you will usually want to run a smaller set of
|
||||
tests. For example, a good "smoke test" that can be used after modifying rustc
|
||||
When working on a specific PR, you will usually want to run a smaller set of tests.
|
||||
For example, a good "smoke test" that can be used after modifying rustc
|
||||
to see if things are generally working correctly would be to exercise the `ui`
|
||||
test suite ([`tests/ui`]):
|
||||
|
||||
|
|
@ -49,9 +49,8 @@ test suite ([`tests/ui`]):
|
|||
./x test tests/ui
|
||||
```
|
||||
|
||||
Of course, the choice of test suites is
|
||||
somewhat arbitrary, and may not suit the task you are doing. For example, if you
|
||||
are hacking on debuginfo, you may be better off with the debuginfo test suite:
|
||||
Of course, the choice of test suites is somewhat arbitrary, and may not suit the task you are doing.
|
||||
For example, if you are hacking on debuginfo, you may be better off with the debuginfo test suite:
|
||||
|
||||
```text
|
||||
./x test tests/debuginfo
|
||||
|
|
@ -77,8 +76,8 @@ Likewise, you can test a single file by passing its path:
|
|||
./x test tests/ui/const-generics/const-test.rs
|
||||
```
|
||||
|
||||
`x` doesn't support running a single tool test by passing its path yet. You'll
|
||||
have to use the `--test-args` argument as described
|
||||
`x` doesn't support running a single tool test by passing its path yet.
|
||||
You'll have to use the `--test-args` argument as described
|
||||
[below](#running-an-individual-test).
|
||||
|
||||
```text
|
||||
|
|
@ -97,8 +96,8 @@ have to use the `--test-args` argument as described
|
|||
./x test --stage 0 library/std
|
||||
```
|
||||
|
||||
Note that this only runs tests on `std`; if you want to test `core` or other
|
||||
crates, you have to specify those explicitly.
|
||||
Note that this only runs tests on `std`;
|
||||
if you want to test `core` or other crates, you have to specify those explicitly.
|
||||
|
||||
### Run the tidy script and tests on the standard library
|
||||
|
||||
|
|
@ -117,8 +116,8 @@ you avoid having to run tests for components you did not change at all.
|
|||
|
||||
<div class="warning">
|
||||
|
||||
Note that bors only runs the tests with the full stage 2 build; therefore, while
|
||||
the tests **usually** work fine with stage 1, there are some limitations.
|
||||
Note that bors only runs the tests with the full stage 2 build;
|
||||
therefore, while the tests **usually** work fine with stage 1, there are some limitations.
|
||||
|
||||
</div>
|
||||
|
||||
|
|
@ -128,8 +127,8 @@ the tests **usually** work fine with stage 1, there are some limitations.
|
|||
./x test --stage 2
|
||||
```
|
||||
|
||||
<div class="warning">
|
||||
You almost never need to do this; CI will run these tests for you.
|
||||
<div class="warning"> You almost never need to do this;
|
||||
CI will run these tests for you.
|
||||
</div>
|
||||
|
||||
## Run unit tests on the compiler/library
|
||||
|
|
@ -140,7 +139,8 @@ You may want to run unit tests on a specific file with following:
|
|||
./x test compiler/rustc_data_structures/src/thin_vec/tests.rs
|
||||
```
|
||||
|
||||
But unfortunately, it's impossible. You should invoke the following instead:
|
||||
But unfortunately, it's impossible.
|
||||
You should invoke the following instead:
|
||||
|
||||
```text
|
||||
./x test compiler/rustc_data_structures/ --test-args thin_vec
|
||||
|
|
@ -149,9 +149,9 @@ But unfortunately, it's impossible. You should invoke the following instead:
|
|||
## Running an individual test
|
||||
|
||||
Another common thing that people want to do is to run an **individual test**,
|
||||
often the test they are trying to fix. As mentioned earlier, you may pass the
|
||||
full file path to achieve this, or alternatively one may invoke `x` with the
|
||||
`--test-args` option:
|
||||
often the test they are trying to fix.
|
||||
As mentioned earlier, you may pass the
|
||||
full file path to achieve this, or alternatively one may invoke `x` with the `--test-args` option:
|
||||
|
||||
```text
|
||||
./x test tests/ui --test-args issue-1234
|
||||
|
|
@ -159,14 +159,13 @@ full file path to achieve this, or alternatively one may invoke `x` with the
|
|||
|
||||
Under the hood, the test runner invokes the standard Rust test runner (the same
|
||||
one you get with `#[test]`), so this command would wind up filtering for tests
|
||||
that include "issue-1234" in the name. Thus, `--test-args` is a good way to run
|
||||
a collection of related tests.
|
||||
that include "issue-1234" in the name.
|
||||
Thus, `--test-args` is a good way to run a collection of related tests.
|
||||
|
||||
## Passing arguments to `rustc` when running tests
|
||||
|
||||
It can sometimes be useful to run some tests with specific compiler arguments,
|
||||
without using `RUSTFLAGS` (during development of unstable features, with `-Z`
|
||||
flags, for example).
|
||||
without using `RUSTFLAGS` (during development of unstable features, with `-Z` flags, for example).
|
||||
|
||||
This can be done with `./x test`'s `--compiletest-rustc-args` option, to pass
|
||||
additional arguments to the compiler when building the tests.
|
||||
|
|
@ -176,8 +175,7 @@ additional arguments to the compiler when building the tests.
|
|||
If you have changed the compiler's output intentionally, or you are making a new
|
||||
test, you can pass `--bless` to the test subcommand.
|
||||
|
||||
As an example,
|
||||
if some tests in `tests/ui` are failing, you can run this command:
|
||||
As an example, if some tests in `tests/ui` are failing, you can run this command:
|
||||
|
||||
```text
|
||||
./x test tests/ui --bless
|
||||
|
|
@ -192,8 +190,9 @@ just like when running the tests without the `--bless` flag.
|
|||
There are a few options for running tests:
|
||||
|
||||
* `bootstrap.toml` has the `rust.verbose-tests` option. If `false`, each test will
|
||||
print a single dot (the default). If `true`, the name of every test will be
|
||||
printed. This is equivalent to the `--quiet` option in the [Rust test
|
||||
print a single dot (the default).
|
||||
If `true`, the name of every test will be printed.
|
||||
This is equivalent to the `--quiet` option in the [Rust test
|
||||
harness](https://doc.rust-lang.org/rustc/tests/).
|
||||
* The environment variable `RUST_TEST_THREADS` can be set to the number of
|
||||
concurrent threads to use for testing.
|
||||
|
|
@ -202,24 +201,24 @@ There are a few options for running tests:
|
|||
|
||||
Pass UI tests now have three modes, `check-pass`, `build-pass` and `run-pass`.
|
||||
When `--pass $mode` is passed, these tests will be forced to run under the given
|
||||
`$mode` unless the directive `//@ ignore-pass` exists in the test file. For
|
||||
example, you can run all the tests in `tests/ui` as `check-pass`:
|
||||
`$mode` unless the directive `//@ ignore-pass` exists in the test file.
|
||||
For example, you can run all the tests in `tests/ui` as `check-pass`:
|
||||
|
||||
```text
|
||||
./x test tests/ui --pass check
|
||||
```
|
||||
|
||||
By passing `--pass $mode`, you can reduce the testing time. For each mode,
|
||||
please see [Controlling pass/fail
|
||||
By passing `--pass $mode`, you can reduce the testing time.
|
||||
For each mode, please see [Controlling pass/fail
|
||||
expectations](ui.md#controlling-passfail-expectations).
|
||||
|
||||
## Running tests with different "compare modes"
|
||||
|
||||
UI tests may have different output depending on certain "modes" that the
|
||||
compiler is in. For example, when using the Polonius mode, a test `foo.rs` will
|
||||
UI tests may have different output depending on certain "modes" that the compiler is in.
|
||||
For example, when using the Polonius mode, a test `foo.rs` will
|
||||
first look for expected output in `foo.polonius.stderr`, falling back to the
|
||||
usual `foo.stderr` if not found. The following will run the UI test suite in
|
||||
Polonius mode:
|
||||
usual `foo.stderr` if not found.
|
||||
The following will run the UI test suite in Polonius mode:
|
||||
|
||||
```text
|
||||
./x test tests/ui --compare-mode=polonius
|
||||
|
|
@ -229,8 +228,8 @@ See [Compare modes](compiletest.md#compare-modes) for more details.
|
|||
|
||||
## Running tests manually
|
||||
|
||||
Sometimes it's easier and faster to just run the test by hand. Most tests are
|
||||
just `.rs` files, so after [creating a rustup
|
||||
Sometimes it's easier and faster to just run the test by hand.
|
||||
Most tests are just `.rs` files, so after [creating a rustup
|
||||
toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain), you
|
||||
can do something like:
|
||||
|
||||
|
|
@ -238,17 +237,19 @@ can do something like:
|
|||
rustc +stage1 tests/ui/issue-1234.rs
|
||||
```
|
||||
|
||||
This is much faster, but doesn't always work. For example, some tests include
|
||||
This is much faster, but doesn't always work.
|
||||
For example, some tests include
|
||||
directives that specify specific compiler flags, or which rely on other crates,
|
||||
and they may not run the same without those options.
|
||||
|
||||
## Running tests on a remote machine
|
||||
|
||||
Tests may be run on a remote machine (e.g. to test builds for a different
|
||||
architecture). This is done using `remote-test-client` on the build machine to
|
||||
architecture).
|
||||
This is done using `remote-test-client` on the build machine to
|
||||
send test programs to `remote-test-server` running on the remote machine.
|
||||
`remote-test-server` executes the test programs and sends the results back to
|
||||
the build machine. `remote-test-server` provides *unauthenticated remote code
|
||||
`remote-test-server` executes the test programs and sends the results back to the build machine.
|
||||
`remote-test-server` provides *unauthenticated remote code
|
||||
execution* so be careful where it is used.
|
||||
|
||||
To do this, first build `remote-test-server` for the remote machine, e.g. for
|
||||
|
|
@ -258,13 +259,12 @@ RISC-V
|
|||
./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
|
||||
```
|
||||
|
||||
The binary will be created at
|
||||
`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy this
|
||||
over to the remote machine.
|
||||
The binary will be created at `./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`.
|
||||
Copy this over to the remote machine.
|
||||
|
||||
On the remote machine, run the `remote-test-server` with the `--bind
|
||||
0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should look
|
||||
like this:
|
||||
0.0.0.0:12345` flag (and optionally `-v` for verbose output).
|
||||
Output should look like this:
|
||||
|
||||
```text
|
||||
$ ./remote-test-server -v --bind 0.0.0.0:12345
|
||||
|
|
@ -273,12 +273,13 @@ listening on 0.0.0.0:12345!
|
|||
```
|
||||
|
||||
Note that binding the server to 0.0.0.0 will allow all hosts able to reach your
|
||||
machine to execute arbitrary code on your machine. We strongly recommend either
|
||||
machine to execute arbitrary code on your machine.
|
||||
We strongly recommend either
|
||||
setting up a firewall to block external access to port 12345, or to use a more
|
||||
restrictive IP address when binding.
|
||||
|
||||
You can test if the `remote-test-server` is working by connecting to it and
|
||||
sending `ping\n`. It should reply `pong`:
|
||||
You can test if the `remote-test-server` is working by connecting to it and sending `ping\n`.
|
||||
It should reply `pong`:
|
||||
|
||||
```text
|
||||
$ nc $REMOTE_IP 12345
|
||||
|
|
@ -287,8 +288,8 @@ pong
|
|||
```
|
||||
|
||||
To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
|
||||
variable then use `x` as usual. For example, to run `ui` tests for a RISC-V
|
||||
machine with the IP address `1.2.3.4` use
|
||||
variable then use `x` as usual.
|
||||
For example, to run `ui` tests for a RISC-V machine with the IP address `1.2.3.4` use
|
||||
|
||||
```text
|
||||
export TEST_DEVICE_ADDR="1.2.3.4:12345"
|
||||
|
|
@ -315,34 +316,34 @@ run "/tmp/work/test1018/a"
|
|||
[...]
|
||||
```
|
||||
|
||||
Tests are built on the machine running `x` not on the remote machine. Tests
|
||||
which fail to build unexpectedly (or `ui` tests producing incorrect build
|
||||
Tests are built on the machine running `x` not on the remote machine.
|
||||
Tests which fail to build unexpectedly (or `ui` tests producing incorrect build
|
||||
output) may fail without ever running on the remote machine.
|
||||
|
||||
## Testing on emulators
|
||||
|
||||
Some platforms are tested via an emulator for architectures that aren't readily
|
||||
available. For architectures where the standard library is well supported and
|
||||
Some platforms are tested via an emulator for architectures that aren't readily available.
|
||||
For architectures where the standard library is well supported and
|
||||
the host operating system supports TCP/IP networking, see the above instructions
|
||||
for testing on a remote machine (in this case the remote machine is emulated).
|
||||
|
||||
There is also a set of tools for orchestrating running the tests within the
|
||||
emulator. Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are
|
||||
set up to automatically run the tests under emulation on GitHub Actions. The
|
||||
following will take a look at how a target's tests are run under emulation.
|
||||
There is also a set of tools for orchestrating running the tests within the emulator.
|
||||
Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are
|
||||
set up to automatically run the tests under emulation on GitHub Actions.
|
||||
The following will take a look at how a target's tests are run under emulation.
|
||||
|
||||
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
|
||||
architecture. Included in the Rust tree are the tools [remote-test-client] and
|
||||
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU architecture.
|
||||
Included in the Rust tree are the tools [remote-test-client] and
|
||||
[remote-test-server] which are programs for sending test programs and libraries
|
||||
to the emulator, and running the tests within the emulator, and reading the
|
||||
results. The Docker image is set up to launch `remote-test-server` and the
|
||||
to the emulator, and running the tests within the emulator, and reading the results.
|
||||
The Docker image is set up to launch `remote-test-server` and the
|
||||
build tools use `remote-test-client` to communicate with the server to
|
||||
coordinate running tests (see [src/bootstrap/src/core/build_steps/test.rs]).
|
||||
|
||||
To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as
|
||||
a "remote" machine. A curious detail here is that the network is shared between
|
||||
the simulator instance and the host macOS, so we can use the local loopback
|
||||
address `127.0.0.1`. Something like the following should work:
|
||||
To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as a "remote" machine.
|
||||
A curious detail here is that the network is shared between
|
||||
the simulator instance and the host macOS, so we can use the local loopback address `127.0.0.1`.
|
||||
Something like the following should work:
|
||||
|
||||
```sh
|
||||
# Build the test server for the iOS simulator:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue