Validate `ignore` and `only` compiletest directive, and add human-readable ignore reasons
This PR adds strict validation for the `ignore` and `only` compiletest directives, failing if an unknown value is provided to them. Doing so uncovered 79 tests in `tests/ui` that had invalid directives, so this PR also fixes them.
Finally, this PR adds human-readable ignore reasons when tests are ignored due to `ignore` or `only` directives, like *"only executed when the architecture is aarch64"* or *"ignored when the operative system is windows"*. This was the original reason why I started working on this PR and #108659, as we need both of them for Ferrocene.
The PR is a draft because the code is extremely inefficient: it calls `rustc --print=cfg --target $target` for every rustc target (to gather the list of allowed ignore values), which on my system takes between 4s and 5s, and performs a lot of allocations of constant values. I'll fix both of them in the coming days.
r? `@ehuss`
compiletest: Don't allow tests with overlapping prefix names
Some tests will delete their output directory before starting. The output directory is based on the test names. If one test is the prefix of another test, then when that test starts, it could try to delete the output directory of the other test with the longer path, or otherwise clash with it while the two tests are trying to create/delete/modify the same directory.
In practice, this manifested as a random error on macOS where two tests were trying to create/delete/create `rustdoc/primitive` and `rustdoc/primitive/no_std`, which resulted in an EINVAL (InvalidInput) error.
This renames some of the offending tests, adds `compiletest-ignore-dir` to prevent compiletest from processing some files, and adds a check to prevent this from happening in the future.
Fixes#109397
Some tests will delete their output directory before starting.
The output directory is based on the test names.
If one test is the prefix of another test, then when that test
starts, it could try to delete the output directory of the other
test with the longer path.
Include executed tests in the build metrics (and use a custom test display impl)
The main goal of this PR is to include all tests executed in CI inside the build metrics JSON files. I need this for Ferrocene, and `@Mark-Simulacrum` expressed desire to have this as well to ensure all tests are executed at least once somewhere in CI.
Unfortunately implementing this required rewriting inside of bootstrap all of the code to render the test output to console. libtest supports outputting JSON instead of raw text, which we can indeed use to populate the build metrics. Doing that suppresses the console output though, and compared to rustc and Cargo the console output is not included as a JSON field.
Because of that, this PR had to reimplement both the "pretty" format (one test per line, with `rust.verbose-tests = true`), and the "terse" format (the wall of dots, with `rust.verbose-tests = false`). The current implementation should have the exact same output as libtest, except for the benchmark output. libtest's benchmark output is broken in the "terse" format, so since that's our default I slightly improved how it's rendered.
Also, to bring parity with libtest I had to introduce support for coloring output from bootstrap, using the same dependencies `annotate-snippets` uses. It's now possible to use `builder.color_for_stdout(Color::Red, "text")` and `builder.color_for_stderr(Color::Green, "text")` across all of bootstrap, automatically respecting the `--color` flag and whether the stream is a terminal or not.
I recommend reviewing the PR commit-by-commit.
r? `@Mark-Simulacrum`
Add note for mismatched types because of circular dependencies
If you have crate A with a dependency on crate B, and crate B with a dev-dependency on A, then you might see "mismatched types" errors on types that seem to be equal. This PR adds a note that explains that the types are different, because crate B is compiled twice, one time with `cfg(test)` and one time without.
I haven't found a good way to create circular dependencies in UI tests, so I abused the incremental tests instead. As a bonus, incremental tests support "cpass" now.
related to https://github.com/rust-lang/rust/issues/22750
Add sanitizer support for modern iOS platforms
asan and tsan generally support iOS, but that previously wasn't configured in rust. This only adds support for the simulator architectures, and arm64 device architecture, not the older 32 bit architectures.
asan and tsan generally support iOS, but that previously wasn't
configured in rust. This only adds support for the simulator
architectures, and arm64 device architecture, not the older 32 bit
architectures.
