Pin bootstrap checksums and add a tool to update it automatically
⚠️⚠️ This is just a proactive hardening we're performing on the build system, and it's not prompted by any known compromise. If you're aware of security issues being exploited please [check out our responsible disclosure page](https://www.rust-lang.org/policies/security). ⚠️⚠️
---
This PR aims to improve Rust's supply chain security by pinning the checksums of the bootstrap compiler downloaded by `x.py`, preventing a compromised `static.rust-lang.org` from affecting building the compiler. The checksums are stored in `src/stage0.json`, which replaces `src/stage0.txt`. This PR also adds a tool to automatically update the bootstrap compiler.
The changes in this PR were originally discussed in [Zulip](https://zulip-archive.rust-lang.org/stream/241545-t-release/topic/pinning.20stage0.20hashes.html).
## Potential attack
Before this PR, an attacker who wanted to compromise the bootstrap compiler would "just" need to:
1. Gain write access to `static.rust-lang.org`, either by compromising DNS or the underlying storage.
2. Upload compromised binaries and corresponding `.sha256` files to `static.rust-lang.org`.
There is no signature verification in `x.py` as we don't want the build system to depend on GPG. Also, since the checksums were not pinned inside the repository, they were downloaded from `static.rust-lang.org` too: this only protected from accidental changes in `static.rust-lang.org` that didn't change the `*.sha256` files. The attack would allow the attacker to compromise past and future invocations of `x.py`.
## Mitigations introduced in this PR
This PR adds pinned checksums for all the bootstrap components in `src/stage0.json` instead of downloading the checksums from `static.rust-lang.org`. This changes the attack scenario to:
1. Gain write access to `static.rust-lang.org`, either by compromising DNS or the underlying storage.
2. Upload compromised binaries to `static.rust-lang.org`.
3. Land a (reviewed) change in the `rust-lang/rust` repository changing the pinned hashes.
Even with a successful attack, existing clones of the Rust repository won't be affected, and once the attack is detected reverting the pinned hashes changes should be enough to be protected from the attack. This also enables further mitigations to be implemented in following PRs, such as verifying signatures when pinning new checksums (removing the trust on first use aspect of this PR) and adding a check in CI making sure a PR updating the checksum has not been tampered with (see the future improvements section).
## Additional changes
There are additional changes implemented in this PR to enable the mitigation:
* The `src/stage0.txt` file has been replaced with `src/stage0.json`. The reasoning for the change is that there is existing tooling to read and manipulate JSON files compared to the custom format we were using before, and the slight challenge of manually editing JSON files (no comments, no trailing commas) are not a problem thanks to the new `bump-stage0`.
* A new tool has been added to the repository, `bump-stage0`. When invoked, the tool automatically calculates which release should be used as the bootstrap compiler given the current version and channel, gathers all the relevant checksums and updates `src/stage0.json`. The tool can be invoked by running:
```
./x.py run src/tools/bump-stage0
```
* Support for downloading releases from `https://dev-static.rust-lang.org` has been removed, as it's not possible to verify checksums there (it's customary to replace existing artifacts there if a rebuild is warranted). This will require a change to the release process to avoid bumping the bootstrap compiler on beta before the stable release.
## Future improvements
* Add signature verification as part of `bump-stage0`, which would require the attacker to also obtain the release signing keys in order to successfully compromise the bootstrap compiler. This would be fine to add now, as the burden of installing the tool to verify signatures would only be placed on whoever updates the bootstrap compiler, instead of everyone compiling Rust.
* Add a check on CI that ensures the checksums in `src/stage0.json` are the expected ones. If a PR changes the stage0 file CI should also run the `bump-stage0` tool and fail if the output in CI doesn't match the committed file. This prevents the PR author from tweaking the output of the tool manually, which would otherwise be close to impossible for a human to detect.
* Automate creating the PRs bumping the bootstrap compiler, by setting up a scheduled job in GitHub Actions that runs the tool and opens a PR.
* Investigate whether a similar mitigation can be done for "download from CI" components like the prebuilt LLVM.
r? `@Mark-Simulacrum`
Make sure FileCheck is copied in the LLVM output directory
The tool, which is needed by parts of our test suite, is built as part of LLVM but is *not* copied to the directory containing the output LLVM binaries. This adds a flag to ensure the binary is copied. This shouldn't add any extra built time, as the flag just installs extra binaries that were already compiled.
This is not strictly needed for the test suite to work (as it also checks `build/$target/llvm/build/bin` for the binary), but it allows deleting the `build/$TARGET/llvm/build` directory (which also contains the intermediary build artifacts) without affecting the test suite, saving disk space.
Make `-Z gcc-ld=lld` work for Apple targets
`-Z gcc-ld=lld` was introduced in #85961. It does not work on Macos because lld needs be either named `ld64` or passed `-flavor darwin` as the first two arguments in order to select the Mach-O flavor. Rust invokes cc (=clang) on Macos for linking which calls `ld` as linker binary and not `ld64`, so just creating an `ld64` binary and modifying the search path with `-B` does not work.
In order to solve this patch does:
* Set the `lld_flavor` for all Apple-derived targets to `LldFlavor::Ld64`. As far as I can see this actually works towards fixing `-Xlinker=rust-lld` as all those targets use the Mach-O object format.
* Copy/hardlink rust-lld to the gcc-ld subdirectory as ld64 next to ld.
* If `-Z gcc-ld=lld` is used and the target lld flavor is Ld64 add `-fuse-ld=/path/to/ld64` to the linker invocation.
Fixes#86945.
The tool, which is needed by parts of our test suite, is built as part
of LLVM but is *not* copied to the directory containing the output LLVM
binaries. This adds a flag to ensure the binary is copied.
This shouldn't add any extra built time, as the flag just installs extra
binaries that were already compiled.
PGO for LLVM builds on x86_64-unknown-linux-gnu in CI
This shows up to 6% less instruction counts with larger - up to 18% - wins on cycles
on multiple benchmarks, and up to 19% wins on the -j1 wall times for rustc self-compilation.
We can afford to spend the extra cycles building LLVM essentially once more for
the x86_64-unknown-linux-gnu CI build today. The builder finishes in around 50
minutes on average, and this adds just 10 more minutes. Given the sizeable
improvements in compiler performance, this is definitely worth it.
bootstrap.py: recognize riscv64 when auto-detect
The architecture auto-detect table has no entry for riscv64 (which rustc
uses riscv64gc for the first part of triplet, assuming it's a generic
Linux distro).
Add it to the table to allow riscv64 systems to bootstrap Rust.
Signed-off-by: Icenowy Zheng <icenowy@aosc.io>
This shows up to 5% less instruction counts on multiple benchmarks, and up to
19% wins on the -j1 wall times for rustc self-compilation.
We can afford to spend the extra cycles building LLVM essentially once more for
the x86_64-unknown-linux-gnu CI build today. The builder finishes in around 50
minutes on average, and this adds just 10 more minutes. Given the sizeable
improvements in compiler performance, this is definitely worth it.
Allow the iOS toolchain to be built on Linux
The iOS toolchain can be built on Linux with minor changes. The compilation will invoke `xcrun` to find the path to the iPhone SDK but a fake `xcrun` executable can be used.
```
#!/bin/sh
echo "/path/to/sdk"
```
The iOS toolchain can then be built and linked with rustup.
```
$ ./x.py build --stage 2 --host x86_64-unknown-linux-gnu \
--target aarch64-apple-ios
$ rustup toolchain link stage1 build/x86_64-unknown-linux-gnu/stage1
```
It's possible to take this toolchain and compile an iOS executable with it. This requires the ld64 linker and an iOS SDK. The ld64 linker can be taken from [cctools](https://github.com/tpoechtrager/cctools-port). A project's .cargo/config can then be edited to use the linker for this target.
```
[target.aarch64-apple-ios]
linker = "/path/to/cctools/bin/arm-apple-darwin-ld"
rustflags = [
"-C",
"""
link-args=
-F/path/to/sdk/System/Library/Frameworks
-L/path/to/sdk/usr/lib
-L/path/to/sdk/usr/lib/system/
-adhoc_codesign
""",
]
```
Upgrade to LLVM 13
Work in progress update to LLVM 13. Main changes:
* InlineAsm diagnostics reported using SrcMgr diagnostic kind are now handled. Previously these used a separate diag handler.
* Codegen tests are updated for additional attributes.
* Some data layouts have changed.
* Switch `#[used]` attribute from `llvm.used` to `llvm.compiler.used` to avoid SHF_GNU_RETAIN flag introduced in https://reviews.llvm.org/D97448, which appears to trigger a bug in older versions of gold.
* Set `LLVM_INCLUDE_TESTS=OFF` to avoid Python 3.6 requirement.
Upstream issues:
* ~~https://bugs.llvm.org/show_bug.cgi?id=51210 (InlineAsm diagnostic reporting for module asm)~~ Fixed by 1558bb80c0.
* ~~https://bugs.llvm.org/show_bug.cgi?id=51476 (Miscompile on AArch64 due to incorrect comparison elimination)~~ Fixed by 81b106584f.
* https://bugs.llvm.org/show_bug.cgi?id=51207 (Can't set custom section flags anymore). Problematic change reverted in our fork, https://reviews.llvm.org/D107216 posted for upstream revert.
* https://bugs.llvm.org/show_bug.cgi?id=51211 (Regression in codegen for #83623). This is an optimization regression that we may likely have to eat for this release. The fix for #83623 was based on an incorrect premise, and this needs to be properly addressed in the MergeICmps pass.
The [compile-time impact](https://perf.rust-lang.org/compare.html?start=ef9549b6c0efb7525c9b012148689c8d070f9bc0&end=0983094463497eec22d550dad25576a894687002) is mixed, but quite positive as LLVM upgrades go.
The LLVM 13 final release is scheduled for Sep 21st. The current nightly is scheduled for stable release on Oct 21st.
r? `@ghost`
Fix compiling other codegen backends when llvm is enabled
Extracted from #81746
Without this change rustbuild will not pass the required linker argument to find libllvm. While other backends likely don't use libllvm, it is necessary to be able to link against rustc_driver as the llvm backend is linked into it.
The architecture auto-detect table has no entry for riscv64 (which rustc
uses riscv64gc for the first part of triplet, assuming it's a generic
Linux distro).
Add it to the table to allow riscv64 systems to bootstrap Rust.
Signed-off-by: Icenowy Zheng <icenowy@aosc.io>
When LLVM_INCLUDE_TESTS is enabled (by default), LLVM requires
Python 3.6 for the lit test runner, otherwise only Python 3.0 is
required.
As we have many docker images using Ubuntu 16.04, which only has
Python 3.5, this avoids the need to install a newer Python version
for them.
The iOS toolchain can be built on Linux with minor changes. The
compilation will invoke `xcrun` to find the path to the iPhone SDK but
a fake `xcrun` executable can be used.
```
#!/bin/sh
echo "/path/to/sdk"
```
The iOS toolchain can then be built and linked with rustup.
```
$ ./x.py build --stage 2 --host x86_64-unknown-linux-gnu \
--target aarch64-apple-ios
$ rustup toolchain link stage1 build/x86_64-unknown-linux-gnu/stage1
```
It's possible to take this toolchain and compile an iOS executable
with it. This requires the ld64 linker and an iOS SDK. The ld64 linker
can be taken from
[cctools](https://github.com/tpoechtrager/cctools-port). A project's
.cargo/config can then be edited to use the linker for this target.
```
[target.aarch64-apple-ios]
linker = "/path/to/cctools/bin/arm-apple-darwin-ld"
rustflags = [
"-C",
"""
link-args=
-F/path/to/sdk/System/Library/Frameworks
-L/path/to/sdk/usr/lib
-L/path/to/sdk/usr/lib/system/
-adhoc_codesign
""",
]
```
Now that Cargo deduplicates diagnostics from different targets, this doesn't flood the console with
duplicate errors.
Note that this doesn't add `--all-targets` in `Builder::cargo` directly because `impl Step for Std`
actually wants to omit `--all-targets` the first time while it's still building libtest.
When passed `--all-targets`, this warns that the option isn't needed, but still continues to compile.
Add x.py option to --force-rerun compiletest tests
This can be used like `./x.py test src/test/ui/abi/ --force-rerun`, and is useful when verifying that newly blessed tests don't change between test runs (such as due to being dependent on the current time or memory layout or RNG), without needing to change the test file or find the right file in `build` to remove.
Rollup of 7 pull requests
Successful merges:
- #85807 (bootstrap: Disable initial-exec TLS model on powerpc)
- #87761 (Fix overflow in rustc happening if the `err_count()` is reduced in a stage.)
- #87775 (Add hint for unresolved associated trait items if the trait has a single item)
- #87779 (Remove special case for statement `NodeId` assignment)
- #87787 (Use `C-unwind` ABI for `__rust_start_panic` in `panic_abort`)
- #87809 (Fix typo in the ptr documentation)
- #87816 (Sync rustc_codegen_cranelift)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Add config.toml options for enabling overflow checks in rustc and std
The names are `overflow-checks` and `overflow-checks-std` and they work similar to `debug-assertions` and `debug-assertions-std`. Once added we can measure how big the performance impact actually is and maybe enable them for CI tests.
Enabling them already makes two ui tests fail:
```
failures:
[ui] ui/parser/item-free-const-no-body-semantic-fail.rs
[ui] ui/parser/item-free-static-no-body-semantic-fail.rs
```
(See #84219 and #87761.)
bootstrap.py: use `git rev-list` for robustness
Use `git rev-list` instead of `git log` to be more robust against
UI changes in git. Also, use the full email address for bors,
because `--author` uses a substring match.
Based on #87513, but is separate because it's less minimal and may require additional manual testing.
~Open questions:~
* ~Should the `merge_base` search also use `--first-parent`?~
* ~Do we exclude non-merge commits from bors? There are a few, and I'm not sure what they have in common. Some of them look like squashes, and some look like they're in rollup branches.~
r? `@jyn514`
`@rustbot` label +A-rustbuild +C-cleanup
rfc3052 followup: Remove authors field from Cargo manifests
Since RFC 3052 soft deprecated the authors field, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information for contributors, we may as well
remove it from crates in this repo.
Ensure `./x.py dist` adheres to `build.tools`
According to `config.toml.example`, the way to produce dist artifacts for both the compiler and a *subset* of tools would be to enable the extended build and manually specify the list of tools to build:
```toml
[build]
extended = true
tools = ["cargo", "rustfmt"]
```
This works as expected for `./x.py build` and `./x.py install`, but *not* for `./x.py dist`. Before this PR `./x.py dist` simply ignored the contents of `build.tools`, building just rustc/rustdoc if `build.extended = false` and all of the tools otherwise. This PR does two things:
* Changes `./x.py dist extended` to only build the tools defined in `build.tools`, if `build.tools` is not empty. The rest of the extended step was refactored to simplify the code.
* Changes how dist jobs for tools are gated: instead of `assert!(builder.config.extended)` to prevent tools from being built with `build.extended = false`, tools are simply built by default depending on `build.extended` and `build.tools`. This also enables to **explicitly** dist tools even with `build.extended = false`.
This PR is best reviewed commit-by-commit.
Fixes#86436
Since RFC 3052 soft deprecated the authors field anyway, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information, we should remove it from
crates in this repo.
bootstrap.py: change `git log` option to indicate desired behavior
When determining which LLVM artifacts to download, bootstrap.py calls: `git log --author=bors --format=%H -n1 -m --first-parent --
src/llvm-project src/bootstrap/download-ci-llvm-stamp src/version`. However, the `-m` option has no effect, per the `git log` help:
> -m
> This option makes diff output for merge commits to be shown in the
> default format. -m will produce the output only if -p is given as
> well. The default format could be changed using log.diffMerges
> configuration parameter, which default value is separate.
Accordingly, this commit removes use of the -m option in favor of ~~`--diff-merges=off`~~ `--no-patch`, since no diff information is needed, and in fact the presence of a diff breaks the command. Tested using git 2.32, this does not change the output of the command.
The motivation for this change is that some patched versions of git change the behavior of the `-m` flag to imply `-p`, rather than to do nothing unless `-p` is passed. These patched versions of git lead to this script not working. Google's corp-provided git is one such example.
Don't treat git repos as non-existent when `ignore_git` is set
The new submodule handling depends on `is_git()` to be accurate to
decide whether it should handle submodules at all or not. Unfortunately,
`is_git()` treated "this directory does not have a git repository" and
"this repository should not be used for SHA/version/commit date info"
the same. This changes it to distinguish the two.
To clarify: ignore_get is set by default whenever channel == "dev", which it is by default whenever you're compiling locally. So basically everyone would hit this, not just people who had explicitly configured ignore_git.
Here's an example of an error this fixes:
```
$ x build
Updating only changed submodules
Submodules updated in 0.01 seconds
Finished dev [unoptimized + debuginfo] target(s) in 0.17s
warning: x.py has made several changes recently you may want to look at
help: consider looking at the changes in `src/bootstrap/CHANGELOG.md`
note: to silence this warning, add `changelog-seen = 2` at the top of `config.toml`
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.16s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building LLVM for x86_64-unknown-linux-gnu
detected home dir change, cleaning out entire build directory
running: "cmake" "/home/joshua/rustc3/src/llvm-project/llvm" "-G" "Ninja" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_TARGETS_TO_BUILD=AArch64;ARM;BPF;Hexagon;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_INCLUDE_BENCHMARKS=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_BINDINGS=OFF" "-DLLVM_ENABLE_Z3_SOLVER=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=48" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_ENABLE_ZLIB=ON" "-DLLVM_ENABLE_LIBXML2=OFF" "-DLLVM_VERSION_SUFFIX=-rust-dev" "-DCMAKE_INSTALL_MESSAGE=LAZY" "-DCMAKE_C_COMPILER=gcc" "-DCMAKE_CXX_COMPILER=g++" "-DCMAKE_ASM_COMPILER=gcc" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_INSTALL_PREFIX=/home/joshua/rustc3/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_BUILD_TYPE=Release"
CMake Error: The source directory "/home/joshua/rustc3/src/llvm-project/llvm" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
thread 'main' panicked at '
command did not execute successfully, got: exit status: 1
build script failed, must exit now', /home/joshua/.local/lib/cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.44/src/lib.rs:885:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
finished in 0.783 seconds
Build completed unsuccessfully in 0:00:01
```
I *believe* this regression was only introduced in https://github.com/rust-lang/rust/pull/87380, not https://github.com/rust-lang/rust/pull/82653. ``@petrochenkov`` can you check that this fixes the issue you encountered in https://github.com/rust-lang/rust/pull/82653#issuecomment-886113679 ?
r? ``@Mark-Simulacrum``
Only look for commits by bors that are merge commits, because
those are the only ones with CI artifacts. Also, use
`--first-parent` to avoid traversing stuff like rollup branches.
Use `git rev-list` instead of `git log` to be more robust against
UI changes in git. Also, use the full email address for bors,
because `--author` uses a substring match.
