Add new `ignore-backends` and `needs-backends` tests annotations
Part of https://github.com/rust-lang/compiler-team/issues/891.
Next step will be to add these annotations in the files where either the output is different based on the codegen (like `asm` tests) or that are known to fail in the GCC backend.
cc `@oli-obk` `@antoyo`
r? `@Kobzol`
bootstrap: Ignore `rust.debuginfo-level-tests` for codegen tests
As dicussed in https://github.com/rust-lang/rust/issues/61117#issuecomment-495587364, codegen tests typically depend on the raw LLVM IR output and are sensitive to debuginfo level. So do not apply `rust.debuginfo-level-tests` for codegen tests.
Before this commit:
$ ./x test --set rust.debuginfo-level-tests=2 tests/codegen --force-rerun
test result: FAILED. 654 passed; 136 failed; 75 ignored; 0 measured; 0 filtered out; finished in 3.22s
After this commit:
$ ./x test --set rust.debuginfo-level-tests=2 tests/codegen --force-rerun
NOTE: ignoring `rust.debuginfo-level-tests=2` for codegen tests
test result: ok. 790 passed; 0 failed; 75 ignored; 0 measured; 0 filtered out; finished in 3.21s
### Run this in CI?
Maybe it will make sense to add this to CI later but I think it is too early to do now before more non-codegen tests work with `rust.debuginfo-level-tests=2`.
bootstrap: Don't trigger an unnecessary LLVM build from check builds
Coming back to r-l/r development after a few weeks away, I found a major regression in my dev workflows: running `x check compiler` (either manually or via rust-analyzer) would have the side-effect of building LLVM, even though that shouldn't be necessary.
For my main build directory this would be a minor annoyance, but for my separate rust-analyzer build directory it's a huge problem because it causes a completely separate build of LLVM, which takes a long time and should be completely unnecessary.
---
After some investigation, I tracked down the problem to the `can_skip_build` check in this code:
3014e79f9c/src/bootstrap/src/core/build_steps/compile.rs (L1382-L1396)
Historically, this would skip the LLVM build for stage 0 check builds. But after the recent stage 0 std redesign and some associated check stage renumbering (e.g. rust-lang/rust#143048), the condition `builder.top_stage == build_stage` is now false, because `top_stage` is 1 (due to the renumbering) but `build_stage` is 0 (because a “stage 1” non-library check build still uses the stage 0 compiler).
---
Because this is a critical contributor roadblock for me, I have tried to fix this in a relatively narrow way. It's possible that all of this surrounding logic could be greatly simplified (especially in light of the stage redesign changes), but I didn't want this fix to be held back by scope creep.
---
(Zulip thread: https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Bootstrap.20incorrectly.20building.20LLVM.20for.20check.20builds/near/528991035)
Add `ToolTarget` to bootstrap
Oh, you thought I'm done with refactoring bootstrap tools? Na-ah, think again! After the failure of https://github.com/rust-lang/rust/pull/143581, `ToolTarget` is back with a vengeance. This time, I implemented the test changes and tool cleanups without forcing these tools to be built with the stage0 compiler.
There are still some small wins though, `LlvmBitcodeLinker` now starts at stage 1, and not stage 2. Cargo should also be ported to this new mode, but I'm leaving that for a follow-up PR.
Hopefully X-th time's the charm 🤞
r? `@jieyouxu`
bootstrap: Detect musl hosts
Currently, all non-Android Linux hosts are assumed to be using glibc. This obviously isn't very portable and will currently result in downloading a stage0 toolchain for glibc even on musl hosts.
There are multiple ways to detect musl somewhat reliably, but the easiest option is to check for the python target, which is exposed in `sys.implementation._multiarch` and has values like "x86_64-linux-gnu" or "powerpc64le-linux-musl".
Currently, all non-Android Linux hosts are assumed to be using glibc.
This obviously isn't very portable and will currently result in
downloading a stage0 toolchain for glibc even on musl hosts.
There are multiple ways to detect musl somewhat reliably, but the
easiest option is to check for the python SOABI config variable, which
has values like "cpython-313-x86_64-linux-gnu" or
"cpython-313-powerpc64-linux-musl".
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
codegen tests typically depend on the raw LLVM IR output and are
sensitive to debuginfo level. So do not apply
`rust.debuginfo-level-tests` for codegen tests.
Before this commit:
$ ./x test --set rust.debuginfo-level-tests=2 tests/codegen --force-rerun
test result: FAILED. 654 passed; 136 failed; 75 ignored; 0 measured; 0 filtered out; finished in 3.22s
After this commit:
$ ./x test --set rust.debuginfo-level-tests=2 tests/codegen --force-rerun
NOTE: ignoring `rust.debuginfo-level-tests=2` for codegen tests
test result: ok. 790 passed; 0 failed; 75 ignored; 0 measured; 0 filtered out; finished in 3.21s
When a snapshot test fails, it only emits a `.pending-snap` file for the first
snapshot assertion that actually failed, because subsequent assertions aren't
executed. That makes it cumbersome to re-bless tests that contain multiple
snapshot assertions.
Copy GCC sources into the build directory even outside CI
It takes ~3.5s on my Linux notebook to perform the copy, but it should only be executed when we actually go build GCC, and that will almost certainly take much longer :) So I think it should be fine. At least we won't be polluting the source directory for local builds.
Fixes: https://github.com/rust-lang/rust/issues/143986
r? `````@nikic`````
Boostrap: add warning on `optimize = false`
I recently came across a bug that can be traced back to the use of `optimize = false` in `bootstrap.toml` in combination with other settings. Following [this](https://rust-lang.zulipchat.com/#narrow/channel/182449-t-compiler.2Fhelp/topic/Missing.20box-related.20symbols.20with.20panic.20.3D.20'abort'/with/528992909) conversation, this PR adds a warning from `bootstrap` when `optimize = false` is used.
I notice that in the same file I edited there are two different styles for warnings (`WARN`, `Warning`). I used `WARNING` because, by happenstance, when testing I got a `WARNING` that I didn't set a change id: let me know if I can unify the styles in the file I edited.
Don't panic if WASI_SDK_PATH not set when detecting compiler
The fedora packaging builds the wasm sysroot outside of the rust build system. Fedora applies a couple of patches related to wasm which I think make this possible. Not panicking seems consistent with the detection logic of other targets when they cannot find cc.
Drop `./x suggest`
This PR removes the current `./x suggest` implementation (rust-lang/rust#109933, rust-lang/rust#106249) and associated docs for several reasons:
1. Primarily, `./x suggest` is another "flow" in bootstrap that incurs extra complexity and more invariants that bootstrap has to maintain. This causes more friction when trying to investigate and fix staging problems. As far as I know, this flow has not been actively maintained in quite a while, and I'm not aware of interest in maintaining it. Bootstrap really could use less implementation complexity with a very limited maintenance bandwidth.
2. The current `./x suggest` implementation "bypasses" the usual stage defaults for the various check/build/test/etc. flows, and it's not really possible to have a stage default because `./x suggest --run` produces a *sequence* of suggestions like [`./x check`, `./x test library/std`, ..] and then tries to run all of them in sequence, based on which files are modified.
3. We've not seen a lot of interest both in using it or extending static/dynamic test suggestions. Last extensions were rust-lang/rust#117961 and rust-lang/rust#120763. I'm not convinced the extra implementation complexity is worth it. This was discussed in:
- [#t-infra/bootstrap > Dropping the current `./x suggest` flow implementation](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Dropping.20the.20current.20.60.2E.2Fx.20suggest.60.20flow.20implementation/with/527456699)
- [#t-compiler > Dropping current `./x suggest` implementation](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Dropping.20current.20.60.2E.2Fx.20suggest.60.20implementation/with/527528696)
Closesrust-lang/rust#109933 (the current implementation is being removed).
Closesrust-lang/rust#143569 (by removing `./x suggest` altogether).
This is quite a bit of implementation complexity, yet it is quite
broken, and we don't have the maintenance bandwidth to address.
Remove the current implementation if only to reduce bootstrap's
implementation complexity; the `suggest` flow comes with its own set of
hacks.
Rename `stable_mir` and `rustc_smir`
This PR only renames the two crate names.
There is no doubt that we want to rename `stable_mir` to `rustc_public`, while it hasn't been discussed yet that what the new name for `rustc_smir` should be.
This PR proposes a new name for `rustc_smir`, that is `rustc_public_shim`, since `rustc_smir` now is mostly a proxy to do calls to rustc queries and the public API of rustc that is invoked by the `rustc_public` crate.
However, I don't think that name is good enough. I hope there would be a way better name.
r? `@oli-obk`
Run bootstrap tests sooner in the `x test` pipeline
With the recently added bootstrap snapshot tests, and in general with our plans to test more things in bootstrap, I feel like the original comment isn't accurate anymore. Recently, on several occasions I had to wait for 40+ minutes of CI just to find out that the bootstrap snapshot tests have failed. I think we should run bootstrap tests towards the beginning instead now.
r? ```@jieyouxu```
Change bootstrap's `tool.TOOL_NAME.features` to work on any subcommand
This is a followup to rust-lang/rust#142379 to make the bootstrap option `tool.TOOL_NAME.features` work on any subcommand instead of just build (so also run/test/...). I also made the `TOOL_NAME` comparisons look at the tool path instead of the tool name to determine to which tool a `TOOL_NAME` refers to, so you can specify tools by path like in other places of the bootstrap (e.g. `tool."tools/miri".features`).
bootstrap/miri: avoid rebuilds for test builds
When building Miri in its own repo, we always build with `--all-targets`:
a009612691/src/tools/miri/miri-script/src/util.rs (L167-L174)
This saves a bunch of time since some of Miri's dependencies get more features enabled by some of Miri's dev-dependencies, and they all get built twice otherwise if you do `cargo build && cargo test` (which is typically what you end up doing inside `./miri test` and also inside `./x test miri`).
This applies the same approach to bootstrap, drastically reducing the edit-compile cycle for Miri work here. :)
Fix command trace
With the recent developments in centralization of command execution, we somehow broke the traces for command execution. This PR fixes that and add trace to stream command execution as well.
r? ````@Kobzol````
clippy: fix test filtering when TESTNAME is empty
Fixes https://github.com/rust-lang/rust/issues/143824. Turns out bootstrap was just fine, the TESTNAME logic in clippy was wrong... I still made this a rustc PR as that's where I did all the debugging.
The bootstrap change is not really related, but it's comment-only so not worth a separate PR... adding the `test_args` is also part of what `prepare_cargo_test` would usually do so let's group the code properly.
Add --compile-time-deps argument for x check
Together with skipping building C++ code in rustc_llvm for check, this reduces the amount of time it takes to do the x check for rust-analyzer analysis from 12m16s to 3m06s when the bootstrap compiler is already downloaded.
Access `wasi_sdk_path` instead of reading environment variable in bootstrap
Small cleanup to remove an environment variable read that we have performed earlier in bootstrap already.
Remove format short command trait
Since we no longer have traces of the vanilla command, and we're already implementing format_short_command for CommandFingerprint, we can use it directly from the fingerprint. This PR removes the standalone format_short_command trait and moves its implementation under CommandFingerprint.
