Optimize `librustc_driver.so` with BOLT
This PR optimizes `librustc_driver.so` on 64-bit Linux CI with BOLT.
### Code
One thing that's not clear yet to me how to resolve is how to best pass a linker flag that we need for BOLT (the second commit). It is currently passed unconditionally, which is not a good idea. We somehow have to:
1) Only pass it when we actually plan to use BOLT. How to best do that? `config.toml` entry? Environment variable? CLI flag for bootstrap? BOLT optimization is done by `opt-dist`, therefore bootstrap doesn't know about it by default.
2) Only pass it to `librustc_driver.so` (see performance below).
Some discussion of this flag already happened on [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Adding.20a.20one-off.20linker.20flag).
### Performance
Latest perf. results can be found [here](https://github.com/rust-lang/rust/pull/102487#issuecomment-1743469053). Note that instruction counts are not very interesting here, there are only regressions on hello world programs. Probably caused by a larger C++ libstd (?).
Summary:
- ✔️ `-1.8%` mean improvement in cycle counts across many primary benchmarks.
- ✔️ `-1.8%` mean Max-RSS improvement.
- ✖️ 34 MiB (+48%) artifact size regression of `librustc_driver.so`.
- This is caused by building `librustc_driver.so` with relocations (which are required for BOLT). Hopefully, it will be [fixed](https://discourse.llvm.org/t/bolt-rfc-a-new-mode-to-rewrite-entire-binary/68674) in the future with BOLT improvements, but now trying to reduce this size increase is [tricky](https://github.com/rust-lang/rust/pull/114649).
- Note that the size of this file was recently reduced in https://github.com/rust-lang/rust/pull/115554 by pretty much the same amount (33 MiB). So the size after this PR is basically the same as it was for the last ~year.
- ✖️ 1.4 MiB (+53%) artifact size regression of `rustc`.
- This is annoying and pretty much unnecessary. It is caused by the way relocations are currently applied in this PR, because they are applied both to `librustc_driver.so` (where they are needed) and for `rustc` (where they aren't needed), since both are built with a single cargo invocation. We might need e.g. some tricks in the bootstrap `rustc` shim to only apply the relocation flag for the shared library and not for `rustc`.
### CI time
CI (try build) got slower by ~5 minutes, which is fine, IMO. It can be further reduced by running LLVM and `librustc_driver` BOLT profile gathering at the same time (now they are gathered separately for LLVM and `librustc_driver`).
r? `@Mark-Simulacrum`
Also CC `@onur-ozkan,` primarily for the bootstrap linker flag issue.
Reorganize `bootstrap/Cargo.toml`
The information here
5b88d659f8/src/bootstrap/Cargo.toml (L55-L59)
was wrong. This PR fixes that and sorts the dependencies in ascending order.
Additionally, I moved the 'features' section above up to make it appear easier.
add `SAFETY` block on the usage of unsafe `getuid`
We pointed out this unsafe usage in #109859, and as a result, we received a fix PR #116476. However, it's important to note that the `libc::getuid()` never actually fails. This PR aims to clarify its safety.
Pass rustc shim flags using environment variable
This PR implements a generalized way of passing of host flags to the `rustc` shim in bootstrap, as proposed [here](https://github.com/rust-lang/rust/pull/116278#discussion_r1346979960).
I tried to implement the bootstrap side using `OsString`, but then I realized that the shim code was using `env::var` before anyway, instead of `env::var_os`, so I just settled on a `String`. The shim side is still general and uses `env::vars_os` now.
I'm not sure if we actually need to do something with the `rustdoc` shim. It *seems* to me that the env. vars passed to it (`RUSTDOC_LINKER`) and (`RUSTDOC_LLD_NO_THREADS`) could just be passed to cargo directly (or rather, the commands that they invoke in the shim could be passed directly). I'm not sure why are they set by the shim.
r? `@onur-ozkan`
CC `@petrochenkov`
Add RUSTFLAGS_BOOTSTRAP to RUSTFLAGS for bootstrap compilation
Adds `RUSTFLAGS_BOOTSTRAP` to `RUSTFLAGS` for bootstrap compilation when `RUSTFLAGS_BOOTSTRAP` exists in the environment. With this PR, `RUSTFLAGS_BOOTSTRAP` will affect every build(as we already do for rustc and std) compiled with stage0 compiler.
Resolves#94234
Note that RUSTFLAGS_BOOTSTRAP should always be added to the end of
RUSTFLAGS to be actually effective (e.g., if we have `-Dwarnings` in
RUSTFLAGS, passing `-Awarnings` from RUSTFLAGS_BOOTSTRAP should override it).
Signed-off-by: onur-ozkan <work@onurozkan.dev>
bootstrap: copy self-contained linking components to `stage0-sysroot`
I hit this issue while trying to bootstrap using a rustc where `rust-lld` is used by default: this was the cause of the failure to profile rustc-perf's bootstrap benchmark in https://github.com/rust-lang/rust/pull/113382.
`stage0-sysroot` currently only has libs and self-contained objects, not the other self-contained linking components yet. Most notably, it does not contain the linker and wrappers that we build, and that rustup distributes.
If you try to bootstrap using the bootstrap compiler's `rust-lld`, it will fail to link std at stage0 because `rust-lld` and the `gcc-ld` wrappers, will not be found in `stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/bin`.
This PR copies the `bin` directory next to the `lib` directory when `rust.lld` is enabled in the config (though maybe it could be done unconditionally, the fact that we need it to link does not necessarily mean that we'd want to build and provide it at stage1).
cc `@Kobzol` who also encountered this issue while using lld during bootstrap.
bootstrap major change detection implementation
The use of `changelog-seen` and `bootstrap/CHANGELOG.md` has not been functional in any way for many years. We often do major/breaking changes but never update the changelog file or the `changelog-seen`. This is an alternative method for tracking major or breaking changes and informing developers when such changes occur.
Example output when bootstrap detects a major change:
otherwise bootstrap will fail to link the stdlib on a target using the
self-contained linker: rust-lld will not be found since it's currently
not in the stage0-sysroot.
warn if source is not either a git clone or a dist tarball
When the repository is downloaded directly via HTTP(as in #115041), builds may fail due to missing submodules.
This PR adds a check that warns people in such cases.
Pass `-jN` from Make to `BOOTSTRAP_ARGS`
Enables the same functionality as `x -jN` in Make by passing the `-jN` arg from Make to the `BOOTSTRAP_ARGS` if it is specified.
