We don't distribute a miri build for beta/stable so it needs to be kept
optional. In the future it likely makes sense to switch the miri
*artifacts* to always be built, but the rustup component to not be
included -- this will avoid some of this pain.
Don't set `is_preview` for clippy and rustfmt
These have been shipped on stable for many years now and it would be very disruptive to ever remove them.
Remove the `-preview` suffix from their dist components.
Based on https://github.com/rust-lang/rust/pull/102565.
These have been shipped on stable for many years now and it would be very disruptive to ever remove them.
Remove the `-preview` suffix from their dist components.
If clang isn't the C compiler used for the UEFI targets, or if the wrong
`--target` is passed to clang, we will get ELF objects in some
rlibs. This will cause problems at link time when trying to compile a
UEFI program that uses any of those objects. Add a check to the dist
step for UEFI targets that reads each rlib with the `object` crate and
fails with an error if any non-COFF objects are found.
This reverts commit 3acb505ee5
(PR #101833).
The changes in this commit caused several bugs or at least
incompatibilies. For now we're reverting this commit and will re-land it
alongside fixes for those bugs.
Use BOLT in CI to optimize LLVM
This PR adds an optimization step in the Linux `dist` CI pipeline that uses [BOLT](https://github.com/llvm/llvm-project/tree/main/bolt) to optimize the `libLLVM.so` library built by boostrap.
Steps:
- [x] Use LLVM 15 as a bootstrap compiler and use it to build BOLT
- [x] Compile LLVM with support for relocations (`-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-q"`)
- [x] Gather profile data using instrumented LLVM
- [x] Apply profile to LLVM that has already been PGOfied
- [x] Run with BOLT profiling on more benchmarks
- [x] Decide on the order of optimization (PGO -> BOLT?)
- [x] Decide how we should get `bolt` (currently we use the host `bolt`)
- [x] Clean up
The latest perf results can be found [here](https://github.com/rust-lang/rust/pull/94381#issuecomment-1258269440). The current CI build time with BOLT applied is around 1h 55 minutes.
Package `rust-docs-json` into nightly components (take 3)
`dist` creates a `rust-docs-json.tar.xz` tarfile. But build-manifest expected it to be named `rust-docs-json-preview.tar.xz`. Change build-manifest to allow the name without the `-preview` suffix.
I haven't actually tested this :( build-manifest is a pain to run locally.
rust-lang/rust#100557 removed the `git-commit-hash` file and replaced it
with `git-commit-info`. However, build-manifest relies on the
`git-commit-hash` file being present, so this adds it back.
fix: use git-commit-info for version information
Fixes#33286.
Fixes#86587.
This PR changes the current `git-commit-hash` file that `./x.py` dist puts in the `rustc-{version}-src.tar.{x,g}z` to contain the hash, the short hash, and the commit date from which the tarball was created, assuming git was available when it was. It uses this for reading the version so that rustc has all the appropriate metadata.
# Testing
Testing this is kind of a pain. I did it with something like
```sh
./x.py dist # ensure that `ignore-git` is `false` in config.toml
cp ./build/dist/rustc-1.65.0-dev-src.tar.gz ../rustc-1.65.0-dev-src.tar.gz
cd .. && tar -xzf rustc-1.65.0-dev-src && cd rustc-1.65.0-dev-src
./x.py build
```
Then, the output of `rustc -vV` with the stage1 compiler should have the `commit-hash` and `commit-date` fields filled, rather than be `unknown`. To be completely sure, you can use `rustc --sysroot` with the stdlib that the original `./x.py dist` made, which will require that the metadata matches.
`dist` creates a `rust-docs-json.tar.xz` tarfile. But build-manifest expected it to be named
`rust-docs-json-preview.tar.xz`. Change build-manifest to allow the name without the `-preview` suffix.
This also adds `rust-docs-json` to the `rust` component. I'm not quite sure why it exists,
but rustup uses it to determine which components are available.
Make the `c` feature for `compiler-builtins` an explicit opt-in
Its build script doesn't support cross-compilation. I tried fixing it, but the cc crate itself doesn't appear to support cross-compiling to windows either unless you use the -gnu toolchain:
```
error occurred: Failed to find tool. Is `lib.exe` installed?
```
Fixes https://github.com/rust-lang/rust/issues/101172.
The build script for `compiler_builtins` doesn't support cross-compilation. I tried fixing it, but the cc crate itself
doesn't appear to support cross-compiling to windows either unless you use the -gnu toolchain:
```
error occurred: Failed to find tool. Is `lib.exe` installed?
```
Rather than trying to fix it or special-case the platforms without bugs,
make it opt-in instead of automatic.
Distribute bootstrap in CI
This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here:
1. Make it faster to build rust from source, both the first time and incrementally
2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information.
There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage.
Helps with https://github.com/rust-lang/rust/issues/94829.
Add `llvm-dis` to the set of tools in `ci-llvm`
The LLVM disassembler is needed for the test introduced in https://github.com/rust-lang/rust/pull/97550.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Distribute rust-docs-json via rustup.
I am not 100% sure on how to treat `rust-json-docs` in `target_host_combination`. I went along with a similar strategy to the one used for `rust-docs`, but looking for guidance there.
- Add a new `bootstrap` component
Originally, we planned to combine this with the `rust-dev` component.
However, I realized that would force LLVM to be redownloaded whenever bootstrap is modified.
LLVM is a much larger download, so split this to get better caching.
- Build bootstrap for all tier 1 and 2 targets
When running `x.py dist`, rustfmt was being allowed to fail when
missing-tools is true. This isn't much of an issue in practice
since other CI jobs will fail if rustfmt fails. This code was just
leftovers from when rustfmt was tracked in toolstate, and this removes
it to make it clear that it no longer works that way.
This builds `src/tools/rust-analyzer/crates/proc-macro-srv-cli` and
ships it as part of Rustc's dist component. This allows rust-analyzer's
proc macro support to work on all rustc versions (stable, beta and
nightly) starting now.
Before:
```
thread 'main' panicked at 'Unable to build RLS', dist.rs:42:9
```
After:
```
thread 'main' panicked at 'Unable to build submodule tool RLS (use `missing-tools = true` to ignore this failure)
note: not all tools are available on all nightlies
help: see https://forge.rust-lang.org/infra/toolstate.html for more information', dist.rs:43:9
```
bootstrap: Allow building individual crates
This aims to be as unintrusive as possible, but did still require adding a new `tail_args` field to all `Rustc` and `Std` steps.
New library and compiler crates are added to the sysroot as they are built, since it's useful to have e.g. just alloc and not std.
Fixes https://github.com/rust-lang/rust/issues/44293.
Fix `x dist rust-dev` on a fresh checkout
Previously, it required you to manually run `x build` first, because it
assumed the LLVM binaries were already present.
This is difficult to support without submodule handling in bootstrap.py, because cargo will refuse
to vendor sources unless it knows the Cargo.toml files of all tools in tree. Moving vendor support
to rustbuild means that rustbuild will be built without vendoring.
Rather than trying to solve this, just remove support altogether and require
people to use `rustc-src` if they want vendoring (or run `cargo vendor` manually).
Make "Assemble stage1 compiler" orders of magnitude faster (take 2)
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
- Panic if we generate a symbolic link in a tarball
- Change install to use copy internally, like in my previous PR
- Change copy to dereference symbolic links, which avoids the previous regression in #96803.
I also took the liberty of fixing `x dist llvm-tools` to work even if you don't call `x build` previously.
This used to take upwards of 5 seconds for me locally. I found that the
culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change
it to use `copy()` internally, which uses hard links instead when
available.
Note that this has a change in behavior: Installing a file will also
change permissions on the source, not just the destination, if hard
links are used.
To avoid changing the behavior on symlinks for existing code, I
introduce a new function `copy_internal` which only dereferences
symlinks when told to do so.
