Reject running `compiletest` self-tests against stage 0 rustc unless explicitly allowed
Currently, in `pr-check-1`, we run `python3 ../x.py test --stage 0 src/tools/compiletest`, which is `compiletest` self-tests against stage 0 rustc. This makes it very annoying for PRs wanting to change target spec JSON format, which `compiletest` depends on for target information, as otherwise `compiletest` would have to know how to handle 2 different target spec JSON formats and know when to pick which.
Instead of doing that, we change `compiletest` self-tests to reject running against stage 0 `rustc` *unless* explicitly allowed with `build.compiletest-allow-stage0=true`. `build.compiletest-allow-stage0` is a proper bootstrap config option in favor of the ad-hoc `COMPILETEST_FORCE_STAGE0` env var. This means that:
- `./x test src/tools/compiletest --stage=0` is not allowed, unless `build.compiletest-allow-stage0=true` is set. In this scenario, `compiletest` self-tests should be expected to fail unless the stage 0 `rustc` as provided is like codegen_cranelift where it's *actually* built from in-tree `rustc` sources.
- In CI, we change `./x test src/tools/compiletest --stage=0` to `./x test src/tools/compiletest --stage=1`, and move it to `pr-check-2`. Yes, this involves building the stage 1 compiler, but `pr-check-2` already has to build stage 1 compiler to test stage 1 library crates.
- Crucially, this means that **`compiletest` is only intended to support one target spec JSON format**, namely the one corresponding to the in-tree `rustc`.
- This should preserve the `compiletest-use-stage0-libtest` UX optimization where changing `compiler/` tree should still not require rebuilding `compiletest` as long as `build.compiletest-use-stage0-libtest=true`, as that should remain orthogonal. This is completely unlike my previous attempt at https://github.com/rust-lang/rust/pull/144563 that tries to do a way more invasive change which would cause the rebuild problem.
Best reviewed commit-by-commit.
---
r? `@Kobzol`
Free disk space on Windows 2025 runners
I've managed to reduce the time deletion takes by:
- Using powershell, which is generally faster for filesystem operations than msys2
- Performing deletions concurrently then waiting for them all to complete
It still takes 2-10 mins but that's not too bad.
Update wasi-sdk to 27.0 in CI
This updates the wasi-sdk used in CI to build release binaries and run CI with. No major motivation beyond keeping things up-to-date and following the development of wasi-sdk.
This updates the wasi-sdk used in CI to build release binaries and run
CI with. No major motivation beyond keeping things up-to-date and
following the development of wasi-sdk.
move uefi test to run-make
Turn the `uefi` test into a more standard `run-make` test, and execute it using the `test-various` CI job like before.
This is just a straightforward translation of the python code, but using `run-make` to supply the target (hence the 3 separate calls in the docker file).
r? ```@jieyouxu```
cc ```@nicholasbishop```
try-job: test-various
Enforce that PR CI jobs are a subset of Auto CI jobs modulo carve-outs
### Background
Currently, it is possible for a PR with red PR-only CI to pass Auto CI, then all subsequent PR CI runs will be red until that is fixed, even in completely unrelated PRs. For instance, this happened with PR-CI-only Spellcheck (rust-lang/rust#144183).
See more discussions at [#t-infra > Spellcheck workflow now fails on all PRs (tree bad?)](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/Spellcheck.20workflow.20now.20fails.20on.20all.20PRs.20.28tree.20bad.3F.29/with/529769404).
### CI invariant: PR CI jobs are a subset of Auto CI jobs modulo carve-outs
To prevent red PR CI in completely unrelated subsequent PRs and PR CI runs, we need to maintain an invariant that **PR CI jobs are a subset of Auto CI jobs modulo carve-outs**.
This is **not** a "strict" subset relationship: some jobs necessarily have to differ under PR CI and Auto CI environments, at least in the current setup. Still, we can try to enforce a weaker "subset modulo carve-outs" relationship between CI jobs and their corresponding Auto jobs. For instance:
- `x86_64-gnu-tools` will have `auto`-only env vars like `DEPLOY_TOOLSTATES_JSON: toolstates-linux.json`.
- `tidy` will want to `continue_on_error: true` in PR CI to allow for more "useful" compilation errors to also be reported, whereas it should be `continue_on_error: false` in Auto CI to prevent wasting Auto CI resources.
The **carve-outs** are:
1. `env` variables.
2. `continue_on_error`.
We enforce this invariant through `citool`, so only affects job definitions that are handled by `citool`. Notably, this is not sufficient *alone* to address the CI-only Spellcheck issue (rust-lang/rust#144183). To carry out this enforcement, we modify `citool` to auto-register PR jobs as Auto jobs with `continue_on_error` overridden to `false` **unless** there's an overriding Auto job for the PR job of the same name that only differs by the permitted **carve-outs**.
### Addressing the Spellcheck PR-only CI issue
Note that Spellcheck currently does not go through `citool` or `bootstrap`, and is its own GitHub Actions workflow. To actually address the PR-CI-only Spellcheck issue (rust-lang/rust#144183), and carry out the subset-modulo-carve-outs enforcement universally, this PR additionally **removes the current Spellcheck implementation** (a separate GitHub Actions Workflow). That is incompatible with Homu unless we do some hacks in the main CI workflow.
This effectively partially reverts rust-lang/rust#134006 (the separate workflow part, not the tidy extra checks component), but is not prejudice against relanding the `typos`-based spellcheck in another implementation that goes through the usual bootstrap CI workflow so that it does work with Homu. The `typos`-based spellcheck seems to have a good false-positive rate.
Closesrust-lang/rust#144183.
---
r? infra-ci
Rename `tests/{assembly,codegen}` into `tests/{assembly,codegen}-llvm` and ignore these testsuites if configured backend doesn't match
Follow-up of https://github.com/rust-lang/rust/pull/144125.
This PR changes `compiletest` so that `asm` tests are only run if they match the current codegen backend. To better reflect it, I renamed the `tests/ui/asm` folder into `tests/ui/asm-llvm`. Like that, we can add new asm tests for other backends if we want without needing to add extra code to `compiletest`.
Next step will be to use the new code annotations added in rust-lang/rust#144125 to ignore ui tests failing in cg_gcc until it's fixed on our side.
cc `@antoyo` `@oli-obk`
r? `@Kobzol`
tidy: move rustdoc js stuff into a tidy extra check
Most of these were factored out of CI scripts, but `eslint` in particular was previously implemented with its own special cased logic.
A new option has been added to bootstrap, `build.tidy-extra-checks`, which serves as a default value for the `--extra-checks` flag. This is mostly for the benefit of rustdoc js maintainers, but should also help bootstrap py maintainers.
Additionally, `--extra-checks=cpp` has been documented.
I'm not super happy with how long the extra check names are in comparison to the others (in particular `typecheck`), but I couldn't think of anything better (I didn't want to name it `tsc` on the off chance we want to switch to a different typechecking engine in the future).
It would be nice to convert the extra checks arg into a proper enum, both for warning on unknown values and to provide better shell completion.
r? ```@GuillaumeGomez```
Fixes: https://github.com/rust-lang/rust/issues/144093
To prevent possibility of a PR with red PR-only CI passing Auto CI, then
all subsequent PR CI runs will be red until that is fixed.
Note that this is **not** a "strict" subset relationship: some jobs
necessarily have to differ under PR CI and Auto CI environments. For
instance:
- `x86_64-gnu-tools` will have auto-only env vars like
`DEPLOY_TOOLSTATES_JSON: toolstates-linux.json`.
- `tidy` will want to `continue_on_error: true` in PR CI to allow for
more "useful" compilation errors to also be reported, whereas it needs
to `continue_on_error: false` in Auto CI to prevent wasting Auto CI
resources.
The carve-outs are:
1. `env` variables.
2. `continue_on_error`.
this makes us less vulnerable to MITM and supply chain attacks.
it also means that the CI scripts are no longer responsible for
tracking the versions of these tools.
it should also avoid the situation where local tsc and CI
disagree on the presense of errors due to them being different versions.
Fix handling of SCRIPT_ARG in docker images
Instead of making this a build parameter, pass the SCRIPT as an environment variable.
To this purpose, normalize on always referring to a script in `/scripts`.
For i686-gnu-nopt-2 I had to create a separate script, because Docker seems to be really terrible at command line argument parsing, so it's not possible to pass an environment variable that contains whitespace.
Fixes https://github.com/rust-lang/rust/issues/143962.
try-job: `dist-x86_64-linux`
try-job: `i686-gnu-nopt-*`
try-job: `i686-gnu-*`
try-job: `x86_64-gnu-llvm-19-*`
try-job: `x86_64-gnu-llvm-20-*`
ci cleanup: rustdoc-gui-test now installs browser-ui-test
this removes the need for --unsafe-perm in the Dockerfile.
cc ```@GuillaumeGomez``` ```@Kobzol```
Instead of making this a build parameter, pass the SCRIPT as an
environment variable.
To this purpose, normalize on always referring to a script in
`/scripts`.
For i686-gnu-nopt-2 I had to create a separate script, because
Docker seems to be really terrible at command line argument
parsing, so it's not possible to pass an environment variable that
contains whitespace.
Fix fallback for CI_JOB_NAME
If CI_JOB_NAME is not specified, it's supposed to fall back to the image name, which is `$image`, not `$IMAGE`.
Failing to set the correct CI_JOB_NAME causes failures when running `dist-ohos-*` images locally.
If CI_JOB_NAME is not specified, it's supposed to fall back to
the image name, which is `$image`, not `$IMAGE`.
Failing to set the correct CI_JOB_NAME causes failures when running
`dist-ohos-*` images locally.
Add an aarch64-msvc build running on ARM64 Windows
Resurrecting rust-lang/rust#126341
Per <https://github.com/rust-lang/rfcs/pull/3817> we intend to promote `aarch64-pc-windows-msvc` to Tier 1. As part of that work, we are adding a pre-merge CI job to validate that changes do not break this target.
Additionally, for consistency, the `dist-aarch64-msvc` job will also be run on Arm64 Windows runners.
r? ``@Kobzol``
try-job: `*aarch64-msvc*`
Get rid of build-powerpc64le-toolchain.sh
The dist-powerpc64le-linux-musl runner never actually used the toolchain that the script produced, it instead used the one from crosstool-ng.
The dist-powerpc64le-linux-gnu runner did use it, from what I can tell mainly to get a glibc 2.17 version with ppc64le support backported. Since crosstool-ng has the necessary patches, we can just use crosstool-ng to get an appropriate toolchain. While at it, use kernel 3.10 headers since that's the version documented in platform support for this target.
try-job: dist-powerpc64le-linux-gnu
try-job: dist-powerpc64le-linux-musl
The dist-powerpc64le-linux-musl runner never actually used the toolchain
that the script produced, it instead used the one from crosstool-ng.
The dist-powerpc64le-linux-gnu runner did use it, from what I can tell
mainly to get a glibc 2.17 version with ppc64le support backported.
Since crosstool-ng has the necessary patches, we can just use
crosstool-ng to get an appropriate toolchain. While at it, use kernel
3.10 headers since that's the version documented in platform support for
this target.
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
Rename `mingw-*` CI jobs to `pr-*`
The name `mingw` confuses people because these CI jobs now do much more than just cross-compile to mingw.
This is basically a find/replace. I chose the name `pr-` because it's job is to do general PR checks,
Skip unnecessary components in x64 try builds
We unnecessarily rebuild `wasm-component-ld`, `llvm-bitcode-linker` and Cranelift during the intermediate PGO builds several times times, which is unnecessarily and increases the duration of try builds. This PR also disables some unnecessary dist components.
r? `````@jieyouxu`````
Enforce in bootstrap that build must have stage at least 1
This PR is a step towards https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523586917. It's very hard or me to make self-contained changes to bootstrap at this moment, so this PR kind of does several things:
1) (first two commits) Try to reduce the usage of `Std::new` in bootstrap, and replace it with a `Builder::std` method (similar to `Builder::compiler`). This is mostly to remove executions of the `Std` step for stage 0, which doesn't make a lot of sense; I'd like to ideally have the invariant that when a step is invoked, it actually builds or does something. Eventually, I'd like for everything to go through `Builder::std`. (Note: I'm not totally married to this idea, if you don't like it, we can remove it from this PR. I mostly did it right now to remove stage 0 std steps from snapshot tests, which shouldn't be there, but we can also filter them out in a different way)
2) Make sure that when you pass `x build compiler`, only the `Assemble` root level step will be invoked, and not the `Rustc` step. Before, both were invoked, which actually ran `Rustc` twice, once with all `crates` filled, and once with no crates (but both actually represent the same situation). Since the `Rustc::make_run` step actually requests a compile that is one stage below it, this actually made `build compiler --stage 0` work, which we don't want to have anymore.
3) Enforce a bootstrap-global invariant that all `build` commands are always on stage `>=1`. If you try to `build` anything on stage 0, it will print a warning and exit bootstrap. This follows the intuition from the new staging rules after the stage redesign; artifacts that are "stage 0" come outside of bootstrap, and we can't really build something for which we don't have source (although we can still test it, but that's for another PR).
Now the logic for build should be quite simple. For pretty much everything except for `Std`, you first use the stage0 compiler to build stage 1. Then you can build a stage 2 <something> using the previously built stage 1 (and then you can continue to stage 3 etc.). And that's it. The default build stage for everything is 1 (modulo download-ci-rustc, but that's a separate can of worms).
The snapshot test infra isn't super nice at the moment, as one of next steps I want to create some simple Builder pattern that will allow us to configure the bootstrap invocations in a more "forward-compatible" way (e.g. now it's not possible to modify the config passed to `configure_with_args`).
There are some things not yet fully resolved for build stage 0:
1) Cargo is still a `ModeRustc` tool, even though it doesn't really have to be, it is buildable with the stage0 compiler
2) bootstrap tools (`opt-dist`, `build-manifest` etc.) are still called stage0 tools, and in the bootstrap output it says something like "stage 0 rustc builds stage 0 opt-dist". Which is a bit weird, but functionally there's no difference, it's just a slightly inconsistent output. We still haven't decided if we should make these tools ignore staging altogether (which is IMO the right choice) or if we want to allow building stage 1/2/3/... bootstrap tools.
r? `@jieyouxu`
try-job: x86_64-rust-for-linux
