pass `CODEGEN_BACKENDS` to docker
The backends to build are now defined in the `CODEGEN_BACKENDS` env var. It's correctly set in CI, but wasn't passed to docker, hence cg_clif wasn't actually built in #81746.
r? `@Kobzol:` I locally tried `CODEGEN_BACKENDS="cranelift" DEPLOY=1 src/ci/docker/run.sh dist-x86_64-linux` and this change was enough for `ci/run.sh` to read the env var.
So I'll try as-is and we'll see.
add bootstrap flag `--skip-stage0-validation`
This change introduces the `--skip-stage0-validation` flag, which permits the use of any desired version of the stage0 compiler without verifying its version.
Additionally, stage0 compiler validation check is reverted(#115103) to its default enabled state.
Helps to #115065
r? Mark-Simulacrum
change default output mode of `BootstrapCommand`
`SuppressOnSuccess` on `BootstrapCommand` is a problematic default mode as it affects the logs during the bootstrapping (as shown in the screenshot below). The default behavior should be to print everything unless we explicitly modify the behavior within build steps.
Fixes#117315
cc `@Kobzol`
Distribute cg_clif as rustup component on the nightly channel
This makes it possible to use cg_clif using:
```bash
$ rustup component add rustc-codegen-cranelift-preview --toolchain nightly
$ RUSTFLAGS="-Zcodegen-backend=cranelift" cargo +nightly build
```
cc https://github.com/rust-lang/compiler-team/issues/405.
r? `@Mark-Simulacrum`
Avoid unnecessary builds/rebuilds of `rust-demangler`
This is a combination of two loosely-related changes:
- Don't build `rust-demangler` as a dependency of `tests/run-make`, because after #112300 none of the remaining run-make tests actually use it. (If future run-make tests ever do need the demangler, it'll be easy to add it back.)
- For `tests/run-coverage`, build the demangler with the stage 0 compiler instead of the current-stage compiler. This avoids having to uselessly rebuild the demangler after modifying and rebuilding the compiler itself.
This avoids needlessly building cg_clif for other targets and makes it
easier for the dist code to determine if it should distribute cg_clif as
component.
The demangler was only needed by coverage tests, but those tests were migrated
into their own custom test mode in #112300.
This avoids having to build the demangler just for run-make tests. It will
still be built as needed by run-coverage tests or for other purposes.
Centralize command running in boostrap (part one)
This PR tries to consolidate the various `run, try_run, run_quiet, run_quiet_delaying_failure, run_delaying_failure` etc. methods on `Builder`. This PR only touches command execution which doesn't produce output that would be later read by bootstrap, and it also only refactors spawning of commands that happens after a builder is created (commands executed during download & git submodule checkout are left as-is, for now).
The `run_cmd` method is quite meaty, but I expect that it will be changing rapidly soon, so I considered it easy to kept everything in a single method, and only after things settle down a bit, then maybe again split it up a bit.
I still kept the original shortcut methods like `run_quiet_delaying_failure`, but they now only delegate to `run_cmd`. I tried to keep the original behavior (or as close to it as possible) for all the various commands, but it is a giant mess, so there may be some deviations. Notably, `cmd.output()` is now always called, instead of just `status()`, which was called previously in some situations.
Apart from the refactored methods, there is also `Config::try_run`, `check_run`, methods that run commands that produce output, oh my… that's left for follow-up PRs :)
The driving goal of this (and following) refactors is to centralize command execution in bootstrap on a single place, to make command mocking feasible.
r? `@onur-ozkan`
Prepare the `bootstrap` tool for the new check-cfg syntax
This PR prepare the `bootstrap` tool for the [new check-cfg syntax](https://github.com/rust-lang/rust/pull/111072) as well as the according [changes to Cargo](https://github.com/rust-lang/cargo/pull/12845).
~~Note that while the new syntax can technically available on stage > 2, we actually cannot use it since we need a cargo version that supports the new syntax which won't happen until the next beta bump (if I understand everything correctly).~~
r? bootstrap
Remap Cargo dependencies to /rust/deps
⚠️ **This doesn't affect user-compiled programs, it only affects building the Rust compiler itself.** ⚠️
Right now, `rust.remap-debuginfo = true` doesn't completely remap all paths: while LLVM and rustc sources are properly remapped (respectively to `/rust/llvm` and `/rust/$commit`), Cargo dependencies still use absolute paths from the Cargo home.
This never affected builds from CI much, because `CARGO_HOME=/cargo` in CI, so users see paths like this included in the precompiled binaries and libraries:
```
/cargo/registry/src/index.crates.io-6f17d22bba15001f/gimli-0.26.2/src/read/line.rs
```
Builds outside CI don't have remapping though, and it's confusing that the config flag doesn't fully do what it advertises.
This PR fixes it by adding remapping for dependencies too. *All registries's* source directory are remapped to `/rust/deps`, to account for multiple registries being able to contain crates.io crates (sparse index vs git, and source replacement mirrors). This results in paths like this being included:
```
/rust/deps/gimli-0.26.2/src/read/line.rs
```
In the bootstrap test, the assertion of the change-id
fails whenever we update the change-id next to a breaking change
in build configurations. This commit removes the assertion,
as it's not critical or useful to have.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
PR #105716 added support for NDK r25b, and removed support for r15. Since
the switch to r25b would have broken existing r15 users anyway, let's
take the opportunity to make the interface more user friendly.
Firstly move the android-ndk property to [build] instead of the
targets. This is possible now that the NDK has obsoleted the concept of
target-specific toolchains.
Also make the property take the NDK root directory instead of the
"toolchains/llvm/prebuilt/<host tag>" subdirectory.
This change introduces the --skip-stage0-validation flag,
which permits the use of any desired version of the stage0
compiler without verifying its version.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Optimize `librustc_driver.so` with BOLT
This PR optimizes `librustc_driver.so` on 64-bit Linux CI with BOLT.
### Code
One thing that's not clear yet to me how to resolve is how to best pass a linker flag that we need for BOLT (the second commit). It is currently passed unconditionally, which is not a good idea. We somehow have to:
1) Only pass it when we actually plan to use BOLT. How to best do that? `config.toml` entry? Environment variable? CLI flag for bootstrap? BOLT optimization is done by `opt-dist`, therefore bootstrap doesn't know about it by default.
2) Only pass it to `librustc_driver.so` (see performance below).
Some discussion of this flag already happened on [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Adding.20a.20one-off.20linker.20flag).
### Performance
Latest perf. results can be found [here](https://github.com/rust-lang/rust/pull/102487#issuecomment-1743469053). Note that instruction counts are not very interesting here, there are only regressions on hello world programs. Probably caused by a larger C++ libstd (?).
Summary:
- ✔️ `-1.8%` mean improvement in cycle counts across many primary benchmarks.
- ✔️ `-1.8%` mean Max-RSS improvement.
- ✖️ 34 MiB (+48%) artifact size regression of `librustc_driver.so`.
- This is caused by building `librustc_driver.so` with relocations (which are required for BOLT). Hopefully, it will be [fixed](https://discourse.llvm.org/t/bolt-rfc-a-new-mode-to-rewrite-entire-binary/68674) in the future with BOLT improvements, but now trying to reduce this size increase is [tricky](https://github.com/rust-lang/rust/pull/114649).
- Note that the size of this file was recently reduced in https://github.com/rust-lang/rust/pull/115554 by pretty much the same amount (33 MiB). So the size after this PR is basically the same as it was for the last ~year.
- ✖️ 1.4 MiB (+53%) artifact size regression of `rustc`.
- This is annoying and pretty much unnecessary. It is caused by the way relocations are currently applied in this PR, because they are applied both to `librustc_driver.so` (where they are needed) and for `rustc` (where they aren't needed), since both are built with a single cargo invocation. We might need e.g. some tricks in the bootstrap `rustc` shim to only apply the relocation flag for the shared library and not for `rustc`.
### CI time
CI (try build) got slower by ~5 minutes, which is fine, IMO. It can be further reduced by running LLVM and `librustc_driver` BOLT profile gathering at the same time (now they are gathered separately for LLVM and `librustc_driver`).
r? `@Mark-Simulacrum`
Also CC `@onur-ozkan,` primarily for the bootstrap linker flag issue.
