This is a companion PR to rust-lang/promote-release#58, which moves the
relevant optimal code to rust-lang/promote-release. As mentioned in the
comments of that PR, this is expected to cut CI costs (and time, though
predominantly felt on fast builders) and reduce wasted resources due to
in-practice single-threaded compression not using the full 8+ vCPU
builders we have available.
Without this, users trying to run `x.py dist` under a sufficiently long
path run into problems when we build the resulting tarballs due to
length limits in the original tar spec. The error looks like:
Finished release [optimized + debuginfo] target(s) in 0.34s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-musl)
Building stage0 tool rust-installer (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.35s
Dist rust-std-1.67.1-x86_64-unknown-linux-musl
Error: failed to generate installer
Caused by:
0: failed to tar file '/home/AAAAAAAAAAAAAA/BBBBBB/CCCC/DDD/EEEEE/FFFFFFFFFFFF/GGGGGGGGGGGGGGGG/HHHHHHHHHH/IIIIIIIIIIIIIII/JJJJJ/KKKKKKK/src/build/tmp/tarball/rust-std/x86_64-unknown-linux-musl/rust-std-1.67.1-x86_64-unknown-linux-musl/rust-std-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/libc.a'
1: provided value is too long when setting link name for
Build completed unsuccessfully in 0:00:03
The fix is to make use of the widely-supported GNU tar extensions which
lift this restriction. Switching to [`tar::Builder::append_link`] takes
care of that for us. See also alexcrichton/tar-rs#273.
[`tar::Builder::append_link`]: https://docs.rs/tar/0.4.38/tar/struct.Builder.html#method.append_link
This moves the rust-installer code to be directly hosted in
rust-lang/rust, since it's not used elsewhere and this makes it easier
to make and review changes without needing a separate upstream commit.
This brings in rust-lang/rust-installer#123, which enables a larger
compression window (8 -> 64MB) amongst other changes to the xz
compression settings. The net effect should be smaller compressed
tarballs which will decrease bandwidth usage for
static.rust-lang.org, download times, and decompression time.
This comes at the cost of higher baseline requirements for running
rustup to use these files, which we believe should be largely acceptable
(running rustc is likely to use at least this much memory) but if we get
specific reports we may explore options to decrease impact (e.g., using
the gzip tarballs automatically in rustup).
This pulls in a fix for the install script on some tr(1) implementations,
as well as an update to use `anyhow` instead of `failure` for error
handling.
This pulls in a commit which uses parallel xz encoding which should
hopefully help shave some time off the dist builders which spend an
inordinate amount of time compressing this data.
This optionally adds lldb (and clang, which it needs) to the build.
Because rust uses LLVM 7, and because clang 7 is not yet released, a
recent git master version of clang is used.
The lldb that is used includes the Rust plugin.
lldb is only built when asked for, or when doing a nightly build on
macOS. Only macOS is done for now due to difficulties with the Python
dependency.
Pull in rust-lang/rust-installer#76 to get streamed tarball generation,
rather than batching it all in memory, while still getting the benefit
of compressing in parallel.
Update `rand` crate to `0.3.19`.
Update `log` crate to `0.3.9` and `0.4.1`.
Update `parking_lot_core` crate to `0.2.9`.
Upgrade all flate2 dependencies to `1.0.1`.
- Update `rust-installer` submodule.
The `copy` function historically in rustbuild used hard links to speed up the
copy operations that it does. This logic was backed out, however, in #39518 due
to a bug that only showed up on Windows, described in #39504. The cause
described in #39504 happened because Cargo, on a fresh build, would overwrite
the previous artifacts with new hard links that Cargo itself manages.
This behavior in Cargo was fixed in rust-lang/cargo#4390 where it no longer
should overwrite files on fresh builds, opportunistically leaving the filesystem
intact and not touching it.
Hopefully this can help speed up local builds by doing fewer copies all over the
place!
It now marks a few whitelisted extensions as executable in the tarball,
so Windows packages can be extracted on other platforms and directly
execute install.sh.
It also includes a fix for the chmod on bulk dirs, so now the html docs
won't be marked executable en masse.
Fixes#42121
r? @alexcrichton
This avoids double compiled Cargo. Hopefully this would speed up (extended) compilation for ~10m.
Notes: when updating Cargo submodule, the replacement version may also need to be updated.
