CI: Update riscv64gc-linux job to Ubuntu 22.04, rename to riscv64gc-gnu
Together with joshua.zivkovic@codethink.co.uk, we've been starting to explore improving the state of the `riscv64gc-unknown-linux-gnu` target. Additionally, I'm looking to add support for this platform in [Ferrocene](https://github.com/ferrocene/ferrocene) ([Related PR](https://github.com/ferrocene/ferrocene/pull/618)).
There currently exists a `src/ci/docker/host-x86_64/disabled/riscv64gc-linux` job for the CI, however it is currently experiencing errors.
<details>
<summary>Errors</summary>
```bash
$ DEPLOY=1 ./src/ci/docker/run.sh riscv64gc-linux
# ...
[RUSTC-TIMING] addr2line test:false 0.371
[RUSTC-TIMING] gimli test:false 3.159
[RUSTC-TIMING] object test:false 4.249
error: linking with `riscv64-linux-gnu-gcc` failed: exit status: 1
|
= note: LC_ALL="C" PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/bin:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/bin:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "riscv64-linux-gnu-gcc" "-Wl,--version-script=/tmp/rustcQaIpWi/list" "-Wl,--no-undefined-version" "/tmp/rustcQaIpWi/symbols.o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/std-ff89a9732cd5d858.std.1b5d59225ff40bd2-cgu.0.rcgu.o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/std-ff89a9732cd5d858.dalhl7sfna1ffn4nhy6pyfa7f.rcgu.rmeta" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/std-ff89a9732cd5d858.ef0znsdf1ihn2bjkmclodhclp.rcgu.o" "-Wl,--as-needed" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/build/compiler_builtins-9e9a40064e2f2bd3/out" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/riscv64gc-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libpanic_unwind-d968371aba64a26c.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libobject-da5b6473912e89d6.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libmemchr-9cfa08d2baa3643e.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libaddr2line-06e0d2153cecb6ce.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libgimli-6fdf5551cec83840.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/librustc_demangle-8ada6466f763fa2e.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libstd_detect-edc0d12d029c4c86.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libhashbrown-9c782935934c8c14.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/librustc_std_workspace_alloc-b6984e43b381efa4.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libminiz_oxide-37ee29bf49ccaa96.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libadler-591133f6804fa0f4.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libunwind-94d98075f42175f3.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libcfg_if-e267a7b9dd7af3a7.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/liblibc-503571a038f8d9fd.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/liballoc-e36c72a5cf0ee45f.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/librustc_std_workspace_core-076c2b8501e25f03.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libcore-c446fff80486d0bb.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libcompiler_builtins-26dc6b5e31e1fdb9.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/riscv64gc-unknown-linux-gnu/lib" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libstd-ff89a9732cd5d858.so" "-shared" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-Wl,-z,origin" "-Wl,-rpath,$ORIGIN/../lib"
# ...
/usr/lib/gcc-cross/riscv64-linux-gnu/9/../../../../riscv64-linux-gnu/bin/ld: failed to merge target specific data of file /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libcompiler_builtins-26dc6b5e31e1fdb9.rlib(compiler_builtins-26dc6b5e31e1fdb9.compiler_builtins.74504a151a6bdbbf-cgu.124.rcgu.o)
/usr/lib/gcc-cross/riscv64-linux-gnu/9/../../../../riscv64-linux-gnu/bin/ld: -march=rv64i2p1_m2p0_a2p1_f2p2_d2p2_c2p0_zicsr2p0: unsupported ISA subset `z'
/usr/lib/gcc-cross/riscv64-linux-gnu/9/../../../../riscv64-linux-gnu/bin/ld: failed to merge target specific data of file /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/riscv64gc-unknown-linux-gnu/release/deps/libcompiler_builtins-26dc6b5e31e1fdb9.rlib(compiler_builtins-26dc6b5e31e1fdb9.compiler_builtins.74504a151a6bdbbf-cgu.004.rcgu.o)
collect2: error: ld returned 1 exit status
[RUSTC-TIMING] std test:false 15.138
error: could not compile `std` (lib) due to 1 previous error
Building bootstrap
Build completed unsuccessfully in 0:04:41
local time: Tue May 28 16:25:09 UTC 2024
network time: Tue, 28 May 2024 16:25:17 GMT
```
</details>
This PR fixes the breakage enough to get the tests running. It does so through bringing the `riscv64gc-unknown-linux-gnu` related test job in line with other related jobs, adopting many of the recent changes present in `src/ci/docker/host-x86_64/armhf-gnu` such as:
* Using Ubuntu 22.04
* Installing a more narrowly scoped package set
* Using `curl` instead of `debootstrap` to set up the root
* No longer patching `busybox`
* Removing the `cmake.sh` script related steps
## Justifying Renaming `riscv64gc-linux` to `riscv64gc-gnu`
The `src/ci/docker/host-x86_64/disabled/riscv64gc-linux` job runs the tests for `risv64gc-unknown-linux-gnu`, it is based off `src/ci/docker/host-x86_64/armhf-gnu`.
There are other jobs that follow a `$arch-gnu` naming scheme:
* `src/ci/docker/host-x86_64/armhf-gnu`
* `src/ci/docker/host-x86_64/x86_64-gnu`
* `src/ci/docker/host-aarch64/aarch64-gnu`
It follows that the name `src/ci/docker/host-x86_64/disabled/riscv64gc-linux` should be `src/ci/docker/host-x86_64/disabled/riscv64gc-gnu`, like the others.
## Testing
> [!NOTE]
> `riscv64gc-unknown-linux-gnu` is a [**Tier 2 with Host Tools** platform](https://doc.rust-lang.org/beta/rustc/platform-support.html), all tests may not necessarily pass! There is work in https://github.com/rust-lang/rust/pull/125220 which helps fix several related tests.
You can test out the renamed job:
```sh
DEPLOY=1 ./src/ci/docker/run.sh riscv64gc-gnu
```
`DEPLOY=1` helps reproduce the CI's environment and also avoids the chance of a `llvm-c/BitReader.h` error (detailed in https://github.com/rust-lang/rust/issues/85424 and https://github.com/rust-lang/rust/issues/56650).
<details>
<summary>Sample of output (expected test failure)</summary>
```bash
$ DEPLOY=1 ./src/ci/docker/run.sh riscv64gc-gnu
# ...
test [ui] tests/ui/where-clauses/where-clause-method-substituion-rpass.rs ... ok
failures:
---- [ui] tests/ui/debuginfo/debuginfo-emit-llvm-ir-and-split-debuginfo.rs stdout ----
error: test compilation failed although it shouldn't!
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/debuginfo/debuginfo-emit-llvm-ir-and-split-debuginfo.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2" "--target=riscv64gc-unknown-linux-gnu" "--check-cfg" "cfg(FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/debuginfo/debuginfo-emit-llvm-ir-and-split-debuginfo" "-A" "unused" "-A" "internal_features" "-Crpath" "-Lnative=/checkout/obj/build/riscv64gc-unknown-linux-gnu/native/rust-test-helpers" "-Clinker=riscv64-linux-gnu-gcc" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/debuginfo/debuginfo-emit-llvm-ir-and-split-debuginfo/auxiliary" "-g" "--emit=llvm-ir" "-Csplit-debuginfo=unpacked"
stdout: none
--- stderr -------------------------------
error: `-Csplit-debuginfo=unpacked` is unstable on this platform
error: aborting due to 1 previous error
------------------------------------------
failures:
[ui] tests/ui/debuginfo/debuginfo-emit-llvm-ir-and-split-debuginfo.rs
test result: FAILED. 5 passed; 1 failed; 16897 ignored; 0 measured; 0 filtered out; finished in 410.99ms
Some tests failed in compiletest suite=ui mode=ui host=x86_64-unknown-linux-gnu target=riscv64gc-unknown-linux-gnu
local time: Tue May 28 16:28:22 UTC 2024
network time: Tue, 28 May 2024 16:28:30 GMT
```
</details>
try-job: riscv64gc-gnu
CI: use a native `arm64` runner
Opening to test if we can use a native GitHub ARM runner instead of the self-hosted one.
The CI has finished [successfully](https://github.com/rust-lang/rust/actions/runs/9414573751/job/25933776975?pr=126113). It took 1h 29m, which is longer than the previous duration (~1h 10m). This is expected, since we currently use 8 cores on GitHub, rather than 14 cores which are configured for the self-hosted runner.
try-job: aarch64-gnu
Weekly `cargo update`
Replaces #125562
`r-efi` needs to be bumped in two places.
The `icu4x` dependencies also added a SPDX license identifer, so remove the license checking exception and add `Unicode-3.0` to the list.
ci: use GCC 13 as cross compiler in `dist-aarch64-linux`
I'm proposing this GCC upgrade since it addresses bug https://github.com/rust-lang/rust/issues/125619. The
regression in question affects stable release consumers who tend to have
no exposure to Rust build tools, so if at all possible, I would like to
have it resolved in the next stable release. I have tried to fix the bug
in `compiler-builtins`, which led to submitting a PR for `compiler-rt`
in upstream LLVM, but it may take a long time before these upstreams
address this regression.
A summary of why upgrading GCC solves the regression follows.
`__multc3()` is a builtin function `compiler-builtins` exposes for
specifically aarch64, non-Windows targets [1]. The object file for it is
included in `staticlib` archives through `libstd`. The implementation
for `__multc3()` is from `multc3.c`, part of LLVM's `compiler-rt`.
Upstream `compiler-rt` normally builds the C file using the Clang
from the same LLVM version. On the other hand, `compiler-builtins`
builds the C file using GCC, outside of the usual LLVM build system.
The upstream implementation doesn't have feature detection which
works for GCC version older than 10, and ends up producing an
unlinkable object.
Upstream LLVM might be slow to respond to this issue as they might deem
`compiler-builtin` as doing something out of the ordinary from their
perspective. They might reasonably assume everyone builds `compiler-rt`
through LLVM's build system.
I have done the following to test this change:
- verified that a local build without this patch exhibits the
regression.
- verified that with this patch, the object for `__multc3()` has no
reference to undefined functions in the symbol table.
- verified that with this patch, `rustc` is usable to build Ruby with
YJIT, and that the reported regression is resolved.
Since `loongarch64-linux-gnu` already uses GCC 13.2.0, I hope we can upgrade without issues.
try-job: dist-aarch64-linux
[1]: c04eb9e1af/build.rs (L524-L539)
It looks like this job was intending to run all of the `needs-matching-clang`
tests (since they don't run without `RUSTBUILD_FORCE_CLANG_BASED_TESTS`), but
over time developed two problems:
- The tests it cares about were moved from run-make-fulldeps to run-make.
- Some of the relevant tests don't actually have "clang" in their name.
Switching to run-make solves the first problem, but we still don't run the
tests without "clang" in their name, because some of them are currently broken.
Add debugging utils and comments to Fuchsia scripts
This should help when debugging a failure in the Fuchsia build in CI.
I plan to follow up with a PR to the testing section of the dev guide with more details, along with more improvements happening in the Fuchsia repo itself.
try-job: x86_64-gnu-integration
Make html rendered by rustdoc allow searching non-English identifier / alias
Fix alias search result showing `undefined` description.
Inspired by https://github.com/rust-lang/mdBook/issues/2393 .
Not sure if it's worth it adding full-text search functionality to rustdoc rendered html.
CI: fix publishing of toolstate history
Hopefully fixes the upload of toolstate history that I broke in https://github.com/rust-lang/rust/pull/125145. The problem is that the toolstate environment variables need to be available not just when updating `latest.json` through the Bash/Python script, but also in the main CI workflow, where `history` is updated in bootstrap (the toolstate logic is distributed in two places :/ ).
The only tricky thing is how to pass the token secret to the CI job, as I need to enable it only for privileged (`auto`/`try`) builds. I assume that the secret is filled only on `rust-lang-ci`, not on `rust-lang`, so I chose the easiest way of just configuring the environment variable globally. We'll see if that works on PR CI.
r? `@ehuss`
Remove unused(?) `~/rustsrc` folder from docker script
Every time I run a docker image, the script creates an empty `~/rustsrc` folder. It doesn't seem to be referenced anywhere else, so I'd like to remove it.
Incidentally, this also documents DIST_TRY_BUILD as I fail to find it from a simple search any time I look for it to eg enable tests on try builds.
I'm proposing this GCC upgrade since it addresses bug #125619. The
regression in question affects stable release consumers who tend to have
no exposure to Rust build tools, so if at all possible, I would like to
have it resolved in the next stable release. I have tried to fix the bug
in `compiler-builtins`, which led to submitting a PR for `compiler-rt`
in upstream LLVM, but it may take a long time before these upstreams
address this regression.
A summary of why upgrading GCC solves the regression follows.
`__multc3()` is a builtin function `compiler-builtins` exposes for
specifically aarch64, non-Windows targets [1]. The object file for it is
included in `staticlib` archives through `libstd`. The implementation
for `__multc3()` is from `multc3.c`, part of LLVM's `compiler-rt`.
Upstream `compiler-rt` normally builds the C file using the Clang
from the same LLVM version. On the other hand, `compiler-builtins`
builds the C file using GCC, outside of the usual LLVM build system.
The upstream implementation doesn't have feature detection which
works for GCC version older than 10, and ends up producing an
unlinkable object.
Upstream LLVM might be slow to respond to this issue as they might deem
`compiler-builtin` as doing something out of the ordinary from their
perspective. They might reasonably assume everyone builds `compiler-rt`
through LLVM's build system.
I have done the following to test this change:
- verified that a local build without this patch exhibits the
regression.
- verified that with this patch, the object for `__multc3()` has no
reference to undefined functions in the symbol table.
- verified that with this patch, `rustc` is usable to build Ruby with
YJIT, and that the reported regression is resolved.
[1]: c04eb9e1af/build.rs (L524-L539)
We want to only demand that we check for all components we expect
if we actually built the components we expect, which means
we built the LLVM. Otherwise, it isn't worth checking.
Add opt-for-size core lib feature flag
Adds a feature flag to the core library that enables the possibility to have smaller implementations for certain algorithms.
So far, the core lib has traded performance for binary size. This is likely what most people want since they have big simd-capable machines. However, people on small machines, like embedded devices, don't enjoy the potential speedup of the bigger algorithms, but do have to pay for them. These microcontrollers often only have 16-1024kB of flash memory.
This PR is the result of some talks with project members like `@Amanieu` at RustNL.
There are some open questions of how this is eventually stabilized, but it's a similar question as with the existing `panic_immediate_abort` feature.
Speaking as someone from the embedded side, we'd rather have this unstable for a while as opposed to not having it at all. In the meantime we can try to use it and also add additional PRs to the core lib that uses the feature flag in areas where we find benefit.
Open questions from my side:
- Is this a good feature name?
- `panic_immediate_abort` is fairly verbose, so I went with something equally verbose
- It's easy to refactor later
- I've added the feature to `std` and `alloc` as well as they might benefit too. Do we agree?
- I expect these to get less usage out of the flag since most size-constraint projects don't use these libraries often.
Upgrade pre-built Clang used in MSVC and MacOS builds, move MSVC builds to Server 2022
Fixes#92948
Example working MacOS and Windows builds: <https://github.com/rust-lang/rust/actions/runs/8989360201>
There is a [bug in Clang 18](https://github.com/llvm/llvm-project/pull/81849) that causes issues when building for Arm64 in later parts of the build (specifically `libgit2`). As a workaround, we will still use the pre-built Clang to build LLVM but will use MSVC for the rest of the Arm64 build.
CI: enable arbitrary try builds, take two
Fixed version of https://github.com/rust-lang/rust/pull/124631, which hopefully won't completely break our CI this time 🤦♂️ Sorry once again. Only the last commit is new.
r? `@pietroalbini`
This commit fixes several issues in the fuchsia-test-runner.py script:
1. Migrate from `pm` to `ffx` for package management, as `pm` is now
deprecated. Furthermore, the `pm` calls used in this script no longer
work at Fuchsia's HEAD. This is the largest change in this commit, and
impacts all steps around repository management (creation and
registration of the repo, as well as package publishing).
2. Allow for `libtest` to be either statically or dynamically linked.
The script assumed it was dynamically linked, but the current Rust
behavior at HEAD is to statically link it.
3. Minor cleanup to use `ffx --machine json` rather than string parsing.
4. Minor cleanup to the docs around the script.
