make it possible to use stage0 libtest on compiletest
With https://github.com/rust-lang/rust/pull/119899, building the library tree will require a stage 1 compiler. This is because `compiletest` is defined as a `ToolStd` (since https://github.com/rust-lang/rust/pull/68019) in order to use the in-tree library. As a result, https://github.com/rust-lang/rust/pull/119899 makes certain development workflows more difficult as changes on the compiler tree will now require recompiling `compiletest` each time.
This PR allows switching `ToolStd` to `ToolBootstrap` with a simple boolean option in `bootstrap.toml` to allow `compiletest` to use the stage 0 `libtest` instead.
The changes under `src/ci` are clearly intended to make sure that `compiletest` doesn't break during future bootstrap beta bumps.
Add bootstrap step diff to CI job analysis
This PR adds another analysis to the job analysis report in GitHub summary. It compares (diffs) bootstrap steps executed by the parent run and by the current commit. This will help us figure out if the bootstrap invocation did something different than before, and also how did the duration of individual steps and bootstrap invocations change.
Can be tested on the https://github.com/rust-lang/rust/pull/119899 PR like this:
```bash
$ curl https://ci-artifacts.rust-lang.org/rustc-builds/3d3394eb64ee2f99ad1a2b849b376220fd38263e/metrics-mingw-check.json > metrics.json
$ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name mingw-check --parent 961351c76c > out.md
```
r? `@marcoie`
Temporarily disable Fuchsia test job to unblock queue
See <https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/fuchsia.20failure/with/506637259> for efforts to fix the test job.
This PR temporarily disables the Fuchsia test job to unblock the queue, so that neither the Fuchsia maintainers nor T-infra maintainers should feel pressured to fix the job ASAP.
Please feel free to re-enable once the test job is fixed.
FYI `@erickt` since you or other Fuchsia maintainers will need to revert this change to merge Fuchsia test job fixes in the future.
r? infra-ci
Only use `DIST_TRY_BUILD` for try jobs that were not selected explicitly
Some CI jobs (x64 Linux, ARM64 Linux and x64 MSVC) use the `opt-dist` tool to build an optimized toolchain using PGO and BOLT. When performing a default try build for x64 Linux, in most cases we want to run perf. on that artifact. To reduce the latency of this common use-case, `opt-dist` skips building several components not needed for perf., and it also skips running post-optimization tests, when it detects that the job is executed as a try job (not a merge/auto job).
This is useful, but it also means that if you *want* to run the tests, you had to go to `jobs.yml` and manually comment this environment variable, create a WIP commit, do a try build, and then remove the WIP commit, which is annoying (in the similar way that modifying what gets run in try builds was annoying before we had the `try-job` annotations).
I thought that we could introduce some additional PR description marker like `try-job-run-tests`, but it's hard to discover that such things exist.
Instead, I think that there's a much simpler heuristic for determining whether `DIST_TRY_BUILD` should be used (that I implemented in this PR):
- If you do just ``@bors` try`, without any custom try jobs selected, `DIST_TRY_BUILD` will be activated, to finish the build as fast as possible.
- If you specify any custom try jobs, you are most likely doing experiments and you want to see if tests pass and everything builds as it should. The `DIST_TRY_BUILD` variable will thus *not* be set in this case.
In this way, if you want to run dist tests, you can just add the `try-job: dist-x86_64-linux` line to the PR description, and you don't need to create any WIP commits.
r? `@marcoieni`
Store test diffs in job summaries and improve analysis formatting
This PR stores the test diffs that we already have in the post-merge workflow also into individual job summaries. This makes it easier to compare test (and later also other) diffs per job, which will be especially useful for try jobs, so that we can actually see the test diffs *before* we merge a given PR.
As a drive-by, I also made a bunch of cleanups in `citool` and in the formatting of the summary and post-merge analyses. These changes are split into self-contained commits.
The analysis can be tested locally with the following command:
```bash
$ curl https://ci-artifacts.rust-lang.org/rustc-builds/<current-sha>/metrics-<job-name>.json > metrics.json
$ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name <job-name> --parent <parent-sha> > out.md
```
For example, for [this PR](https://github.com/rust-lang/rust/pull/138523):
```bash
$ curl https://ci-artifacts.rust-lang.org/rustc-builds/282865097d138c7f0f7a7566db5b761312dd145c/metrics-aarch64-gnu.json > metrics.json
$ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name aarch64-gnu --parent d9e5539a39 > out.md
```
Best reviewed commit by commit.
r? `@marcoieni`
try-job: aarch64-gnu
try-job: dist-x86_64-linux
Update sccache to 0.10.0
This time, does it also for Windows and macOS. This unifies the sccache version across all OSes that we use.
r? `@ghost`
try-job: dist-aarch64-apple
try-job: dist-x86_64-apple
try-job: dist-x86_64-msvc
try-job: dist-x86_64-msvc-alt
try-job: dist-i686-msvc
try-job: dist-aarch64-msvc
try-job: dist-x86_64-linux
try-job: dist-x86_64-netbsd
Improve post-merge workflow
Contains various fixes for the post-merge workflow implemented in https://github.com/rust-lang/rust/pull/138013, which were suggested on Zulip. This PR changes the grouping of test diffs and ignores doctests, as they are too noisy.
I'll post an example output (before/after this PR) in comments below.
r? ```@jieyouxu```
Pass `CI_JOB_DOC_URL` to Docker
Fix-up for https://github.com/rust-lang/rust/pull/136911. I always forget to pass new environment variables to Docker images.. 🤦♂️
r? `@marcoieni`
try-job: x86_64-fuchsia
Enable metrics and verbose tests in PR CI
When debugging CI, I relatively often need to examine what tests are executed on PR CI, and what bootstrap steps does it execute. However, we currently disable both verbose tests and bootstrap metrics on PR CI for some reason.
I'm not actually sure why though, as the PR that (probably) introduced this behavior (https://github.com/rust-lang/rust/pull/51367) didn't explain why.
CC `@oli-obk`
