Dist build manifest
This PR makes two changes that should remove a significant chunk of the time spent in our release process: cloning the `rust-lang/rust` monorepo, all its submodules, and building `bootstrap` to then invoke `build-manifest`:
* `build-manifest` doesn't rely on a clone of the monorepo being present anymore. The only remaining bit of information it fetched from it (the Rust version) is instead bundled in the binary.
* A new "component" is added, `build-manifest`. That component includes a prebuilt version of the tool, and it's *not* included in the Rustup manifest. This will allow `promote-release` to directly invoke the tool without interacting with our build system.
* The Linux x86_64 CI is changed to also build the component mentioned above. It's the only CI builder tasked to do so, and to cleanly support this a new `--include-default-paths` flag was added to `./x.py`.
* The `BUILD_MANIFEST_NUM_THREADS` environment variable is added to configure the number of threads at runtime.
This PR is best reviewed commit-by-commit.
r? `@Mark-Simulacrum`
ci: Fix riscv64gc linux test QEMU fault, plus doc link fix
Newer versions of the `qemu` package (used for riscv64gc-unknown-linux-gnu testing) don't work with the version of the RISC-V bootloader we were using. a4a0342cf5 bumps to a revision which should fix the problem.
e0b033e965 fixes a documentation failure I encountered while running the tests.
Promote aarch64-pc-windows-msvc to Tier 2 Development Platform
Adds a GitHub Actions CI build for `aarch64-pc-windows-msvc` via cross-compilation on an x86_64 host.
This promotes `aarch64-pc-windows-msvc` from a Tier 2 Compilation Target (std) to a Tier 2 Development Platform (std+rustc+cargo+tools).
Fixes#72881
r? `@pietroalbini`
This bumps the version of the bbl bootloader not to perform 64-bit
accesses to the PLIC. Doing so resulted in the QEMU test machine to fail
to boot:
bbl loader
../machine/mtrap.c:21: machine mode: unhandlable trap 7 @ 0x0000000080001f6e
Power off
Signed-off-by: Tom Eccles <tom.eccles@codethink.co.uk>
Stop running macOS builds on Azure Pipelines
The Infrastructure Team agreed to migrate macOS builds to GitHub Actions, so this commit stops running those builders on Azure Pipelines. The GitHub Actions runners are already configured to upload to the production bucket.
We can't still fully remove the Azure Pipelines configuration, as we still need to have that available until no stable releases run any of their builds on Azure Pipelines anymore. I'll open an issue to track fully removing our Azure Pipelines setup once the PR is merged.
r? @Mark-Simulacrum
Ensure that all LLVM components requested by tests are available on CI
Addresses https://github.com/rust-lang/rust/pull/75064#issuecomment-667722652
I used an environment variable because passing a command line option all the way from CI to compiletest would be just too much hassle for this task.
I added a new variable, but any of the already existing ones defined by CI could be used instead.
r? @Mark-Simulacrum
Previously we would have some platforms where LLVM was linked to rustc
statically, but to the LLVM tools dynamically. That meant we were distributing
two copies of LLVM: one as a separate dylib and one statically linked in to
librustc_driver.
Add aarch64-unknown-linux-musl host builds
This adds aarch64-unknown-linux-musl to the hosts list and adds the build to the dist-arm-linux builder as `@Mark-Simulacrum` suggested to me in Zulip. `@jyn514` requested to be mentioned 😄
I had to update the config for crosstool-ng as it had a prompt about the glibc version.
I ran `src/ci/docker/run.sh dist-arm-linux` to test it.
```
Build completed successfully in 1:31:50
Compile requests 8180
Compile requests executed 8135
Cache hits 287
Cache misses 7848
Cache timeouts 0
Cache read errors 0
Forced recaches 0
Cache write errors 0
Compilation failures 0
Cache errors 0
Non-cacheable compilations 0
Non-cacheable calls 36
Non-compilation calls 9
Unsupported compiler calls 0
Average cache write 0.000 s
Average cache read miss 6.389 s
Average cache read hit 0.000 s
Cache location Local disk: "/sccache"
Cache size 202 MiB
Max cache size 10 GiB
== clock drift check ==
local time: Sun Sep 6 19:30:17 UTC 2020
network time: Sun, 06 Sep 2020 19:30:17 GMT
== end clock drift check ==
```
Only errors were in miri due to struct fields being private (already been reported [here](https://github.com/rust-lang/rust/issues/76337))
Edit: Maybe it is helpful if I add that it is a working compiler
```sh
/rust-nightly-aarch64-unknown-linux-musl # ash install.sh
install: creating uninstall script at /usr/local/lib/rustlib/uninstall.sh
install: installing component 'rustc'
install: installing component 'cargo'
install: installing component 'rls-preview'
install: installing component 'rust-analyzer-preview'
install: installing component 'clippy-preview'
install: installing component 'rustfmt-preview'
install: installing component 'llvm-tools-preview'
install: installing component 'rust-analysis-aarch64-unknown-linux-musl'
install: installing component 'rust-std-aarch64-unknown-linux-musl'
install: WARNING: failed to run ldconfig. this may happen when not installing as root. run with --verbose to see the error
Rust is ready to roll.
/ # cat test.rs
fn main() { println!("hello world"); }
/ # rustc test.rs
/ # ./test
hello world
# file test
test: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped
```
Windows doesn't quite support dynamic linking to LLVM yet, but on other
platforms we do. In #76708, it was discovered that we dynamically link to LLVM
from the LLVM tools (e.g., rust-lld), so we need the shared LLVM library to link
against. That means that if we do not have a shared link to LLVM, and want LLVM
tools to work, we'd be shipping two copies of LLVM on all of these platforms:
one in librustc_driver and one in libLLVM.
Also introduce an error into rustbuild if we do end up configured for shared
linking on Windows.
While waiting for a PR job to start testing my code, I noticed the
symlink-build-dir step took 10 minutes to complete, so I investigated
what caused that.
It seems like something changed in the build environment between version
20200901.1 (where the step took 45 seconds) and version 20200908.1
(where the step took 10 minutes). At the time of writing this commit,
the rust-lang organization is on vertsion 20200908.1, while the
rust-lang-ci organization is at version 20200901.1 (and is not affected
by this yet).
There is no need for this step anymore on GHA, as our XL builders got an
increase in the root paritition size, so this commit removes the code
that moved stuff around on GHA (while keeping it on Azure).
For the record, at the time of writing this, the disk situation is:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 667G 60G 607G 9% /
/dev/sdb1 110G 4.1G 101G 4% /mnt
Set ninja=true by default
Ninja substantially improves LLVM build time. On a 96-way system, using
Make took 248s, and using Ninja took 161s, a 35% improvement.
We already require a variety of tools to build Rust. If someone wants to
build without Ninja (for instance, to minimize the set of packages
required to bootstrap a new target), they can easily set `ninja=false`
in `config.toml`. Our defaults should help people build Rust (and LLVM)
faster, to speed up development.
Build dist-x86_64-musl with --enable-profiler.
Trying to build a Rust project with `-Zprofile` for target
x86_64-unknown-linux-musl using rustc 1.46.0-nightly (346aec9b0
2020-07-11), installed with rustup, results in the following error.
```
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"export CARGO_INCREMENTAL=0$ cargo build --target=x86_64-unknown-linux-muslCompiling hello_world v0.1.0 (…)error[E0463]: can't find crate for `profiler_builtins`
|
= note: the compiler may have been built without the profiler runtime
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.error: could not compile `hello_world`.
To learn more, run the command again with --verbose.
```
`-Zprofile` is required here to enable grcov profiling.
This is similar in nature to issue
https://github.com/rust-lang/rust/issues/57257, which has been fixed in
asimilar way at https://github.com/rust-lang/rust/pull/60476 .
A fix for Android has also landed not long ago:
https://github.com/rust-lang/rust/pull/70054 .
Signed-off-by: Tiago Lam <tiagol@hadean.com>
Trying to build a Rust project with `-Zprofile` for target
x86_64-unknown-linux-musl using rustc 1.46.0-nightly (346aec9b0
2020-07-11), installed with rustup, results in the following error.
```
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"export CARGO_INCREMENTAL=0$ cargo build --target=x86_64-unknown-linux-muslCompiling hello_world v0.1.0 (…)error[E0463]: can't find crate for `profiler_builtins`
|
= note: the compiler may have been built without the profiler runtime
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.error: could not compile `hello_world`.
To learn more, run the command again with --verbose.
```
`-Zprofile` is required here to enable grcov profiling.
This is similar in nature to issue
https://github.com/rust-lang/rust/issues/57257, which has been fixed in
asimilar way at https://github.com/rust-lang/rust/pull/60476 .
A fix for Android has also landed not long ago:
https://github.com/rust-lang/rust/pull/70054 .
Signed-off-by: Tiago Lam <tiagol@hadean.com>
This Emscripten version was the first to be cut after the LLVM 11
release branch was created, so it should be the most compatible with
LLVM 11. The old version we were using was incompatible with LLVM 11
because its wasm-ld did not understand all the relocations that LLVM
11 emits.
Move CloudABI to tier 3.
The CloudABI target hasn't had much work done on it in a while, and it doesn't appear to be in active use. It has a fairly substantial amount of code, particularly in the [sys module](5addb135ed/library/std/src/sys/cloudabi) that requires actively supporting. I contacted @EdSchouten who indicated that many of the CloudABI concepts are now in WASI, and that they are OK with the target being moved to tier 3.
