The format of the tarballs produced by CI is roughly the following:
{component}-{release}-{target}.{ext}
While on the beta and nightly channels `{release}` is just the channel
name, on the stable channel is either the Rust version or the version of
the component we're shipping:
cargo-0.47.0-x86_64-unknown-linux-gnu.tar.xz
clippy-0.0.212-x86_64-unknown-linux-gnu.tar.xz
llvm-tools-1.46.0-x86_64-unknown-linux-gnu.tar.xz
miri-0.1.0-x86_64-unknown-linux-gnu.tar.xz
rls-1.41.0-x86_64-unknown-linux-gnu.tar.xz
rust-1.46.0-x86_64-unknown-linux-gnu.tar.xz
...
This makes it really hard to get the package URL without having access
to the manifest (and there is no manifest on ci-artifacts.rlo), as there
is no consistent version number to use.
This commit addresses the problem by always using the Rust version
number as `{release}` for the stable channel, regardless of the version
number of the component we're shipping. I chose that instead of "stable"
to avoid breaking the URL scheme *that* much.
Rustup should not be affected by this change, as it fetches the URLs
from the manifest. Unfortunately we don't have a way to test other
clients before making a stable release, as this change only affects the
stable channel.
Add c as a shorthand check alternative for new options #77603
There is a missing "c" that is a shorthand for "check" in newly added match arm for handling check-specific options.
Fix suggestions for x.py setup
#76631 introduced a new `setup` command to x.py
By default the command prompts for a profile to use:
```
Welcome to the Rust project! What do you want to do with x.py?
a) Contribute to the standard library
b) Contribute to the compiler
c) Contribute to the compiler, and also modify LLVM or codegen
d) Install Rust from source
```
and then displays command suggestions, depending on which profile was chosen. However [the mapping between chosen profile](9cba260df0/src/bootstrap/setup.rs (L75-L85)) and [suggestion](9cba260df0/src/bootstrap/setup.rs (L42-L47)) isn't exact, leading to suggestions not being shown if the user presses `c` or `d`. (because "c" is translated to "llvm" and "d" to "maintainer", but suggestions trigger for "codegen" and "user" respectively)
A more thorough refactor would stop using "strings-as-type" to make sure this kind of error doesn't happen, but it may be overkill for that kind of "script" program?
Tagging the setup command author: @jyn514
Add a command to install a git hook to automatically run `x.py test tidy --bless`
Some folks (such as myself) would probably find a lot of convenience in a pre-commit hook that automatically runs tidy before committing, to avoid burning CI time learning that your commit wasn't tidy.
I'm absolutely positive I have missed some stuff. I basically just got this to where you can run `./x.py run install-git-hook` and then clicked the commit button. Please let me know what else you'd like me to add before this can be merged!
[rustc-dev-guide companion PR](https://github.com/rust-lang/rustc-dev-guide/pull/848)
Rename bootstrap/defaults/{config.toml.PROFILE => config.PROFILE.toml}
This allows these files to have okay syntax highlighting in editors, and helps avoid nagging from editors which want to suggest that I install a plugin for `*.library` files to view the `config.toml.library` or whatever.
It's a very minor change.
r?@jyn514
Improve build-manifest to work with the improved promote-release
This PR makes some changes to build-manifest to have it work better with the other improvements I'm making to [promote-release](https://github.com/rust-lang/promote-release).
A new way to invoke the tool was added: `./x.py run src/tools/build-manifest`. The new invocation disables the generation of `.sha256` files and the generation of GPG signatures, as those steps are not tied to the Rust version we're building the manifest of: handling them in `promote-release` will improve the maintenability of our release process. Invocations through the old command (`./x.py dist hash-and-sign`) are referred inside the source code as "legacy". The new invocation also enables internal parallelism, disabled on legacy to avoid overloading our old server.
Improvements were also made on how the checksums included in the manifest are generated:
* The manifest is first generated with placeholder checksums, and then a function walks through the manifes and calculates only the needed hashes. Before this PR, all the hashes were calculated beforehand, including the hashes of unused files.
* Calculating the hashes is now done in parallel with rayon, to better utilize all the available disk bandwidth.
* The `sha2` crate is now used instead of the `sha256sum` CLI tool: this avoids the overhead of calling another process, but more importantly enables hardware acceleration whenever available (the `sha256sum` CLI tool doesn't support it at all).
r? @Mark-Simulacrum
This PR is best reviewed commit-by-commit.
This matches Cargo behavior and avoids the (somewhat expensive) double checking,
as well as the unfortunate duplicate error messages (#76822,
rust-lang/cargo#5128).
Remove --cfg dox from rustdoc.rs
This was added in https://github.com/rust-lang/rust/pull/53076 because
several dependencies were using `cfg(dox)` instead of `cfg(rustdoc)` (now `cfg(doc)`).
I ran `rg 'cfg\(dox\)'` on the source tree with no matches, so I think
this is now safe to remove.
r? `@Mark-Simulacrum`
cc `@QuietMisdreavus` :)
Don't warn if the config file is somewhere other than `config.toml`
Previously, `config.config` was always hardcoded as `"config.toml"`.
I thought that it was being overridden with the actual value later, but
it turns out `flags.config` was being completely discarded. This keeps
`config.config` in sync with `flags.config`.
Fixes https://github.com/rust-lang/rust/issues/77293
r? `@Mark-Simulacrum`
cc `@davidtwco`
bootstrap: Always build for host, even when target is given
This changes the behavior from *not* building for host whenever an
explicit target is specified. I find this much less confusing.
You can still disable host steps by passing an explicit empty list for
host.
Fixes#76990.
r? `@Mark-Simulacrum`
This was added in https://github.com/rust-lang/rust/pull/53076 because
several dependencies were using `cfg(dox)` instead of `cfg(rustdoc)`.
I ran `rg 'cfg\(dox\)'` on the source tree with no matches, so I think
this is now safe to remove.
Refactor versions detection in build-manifest
This PR refactors how `build-manifest` handles versions, making the following changes:
* `build-manifest` now detects the "package releases" on its own, without relying on rustbuild providing them through CLI arguments. This drastically simplifies calling the tool outside of `x.py`, and will allow to ship the prebuilt tool in a tarball in the future, with the goal of stopping to invoke `x.py` during `promote-release`.
* The `tar` command is not used to extract the version and the git hash from tarballs anymore. The `flate2` and `tar` crates are used instead. This makes detecting those pieces of data way faster, as the archive is decompressed just once and we stop parsing the archive once all the information is retrieved.
* The code to extract the version and the git hash now stores all the collected data dynamically, without requiring to add new fields to the `Builder` struct every time.
I tested the changes locally and it should behave the same as before.
r? `@Mark-Simulacrum`
This changes the behavior from *not* building for host whenever an
explicit target is specified. I find this much less confusing.
You can still disable host steps by passing an explicit empty list for
host.
Fixes#76990.
Previously, `config.config` was always hardcoded as `"config.toml"`.
I thought that it was being overridden with the actual value later, but
it turns out `flags.config` was being completely discarded. This keeps
`config.config` in sync with `flags.config`.
LVI hardening tests
Mitigating the speculative execution LVI attack against SGX enclaves requires compiler changes (i.e., adding lfences). This pull requests adds various tests to check if this happens correctly.
Add `x.py setup`
Closes#76503.
- Suggest `x.py setup` if config.toml doesn't exist yet
- Prompt for a profile if not given on the command line
- Print the configuration that will be used
- Print helpful starting commands after setup
- Link to the dev-guide after finishing
Install std for non-host targets
It seems reasonable that when configuring various targets you'd expect all of them to get std installed, even if you're not building compiler toolchains for each of those.
cc #76990
r? @alexcrichton
Invalidate local LLVM cache less often
This avoids a download of LLVM after every rebase. The downside to this is that if we land some patch affecting LLVM built in CI that breaks this option, but that PR does not update the LLVM submodule, we'll likely not notice until the next update -- but this seems unlikely to happen in practice and I am not personally worried about it.
r? @alexcrichton
Add `--keep-stage-std` to `x.py` for keeping only standard library artifacts
Unlike `--keep-stage 0`, `--keep-stage-std 0` will allow the stage 0 compiler artifacts (i.e., stage1/bin/rustc) to be rebuilt if it has changed. This allows contributors to iterate on later stages of the compiler in tandem with the standard library without needing to to rebuild the entire compiler. I often run into this when working on const-checking, since I may need to add a feature gate or make a small tweak to the standard library.
- Suggest `x.py setup` if config.toml doesn't exist yet (twice, once
before and once after the build)
- Prompt for a profile if not given on the command line
- Print the configuration file that will be used
- Print helpful starting commands after setup
- Link to the dev-guide after finishing
- Note that distro maintainers will see the changelog warning
This keeps only the `std` artifacts compiled by the given stage, not the
compiler. This is useful when working on the latter stages of the
compiler in tandem with the standard library, since you don't have to
rebuild the *entire* compiler when the standard library changes.
