bootstrap: Always build for host, even when target is given
This changes the behavior from *not* building for host whenever an
explicit target is specified. I find this much less confusing.
You can still disable host steps by passing an explicit empty list for
host.
Fixes#76990.
r? `@Mark-Simulacrum`
This was added in https://github.com/rust-lang/rust/pull/53076 because
several dependencies were using `cfg(dox)` instead of `cfg(rustdoc)`.
I ran `rg 'cfg\(dox\)'` on the source tree with no matches, so I think
this is now safe to remove.
Refactor versions detection in build-manifest
This PR refactors how `build-manifest` handles versions, making the following changes:
* `build-manifest` now detects the "package releases" on its own, without relying on rustbuild providing them through CLI arguments. This drastically simplifies calling the tool outside of `x.py`, and will allow to ship the prebuilt tool in a tarball in the future, with the goal of stopping to invoke `x.py` during `promote-release`.
* The `tar` command is not used to extract the version and the git hash from tarballs anymore. The `flate2` and `tar` crates are used instead. This makes detecting those pieces of data way faster, as the archive is decompressed just once and we stop parsing the archive once all the information is retrieved.
* The code to extract the version and the git hash now stores all the collected data dynamically, without requiring to add new fields to the `Builder` struct every time.
I tested the changes locally and it should behave the same as before.
r? `@Mark-Simulacrum`
This changes the behavior from *not* building for host whenever an
explicit target is specified. I find this much less confusing.
You can still disable host steps by passing an explicit empty list for
host.
Fixes#76990.
Previously, `config.config` was always hardcoded as `"config.toml"`.
I thought that it was being overridden with the actual value later, but
it turns out `flags.config` was being completely discarded. This keeps
`config.config` in sync with `flags.config`.
LVI hardening tests
Mitigating the speculative execution LVI attack against SGX enclaves requires compiler changes (i.e., adding lfences). This pull requests adds various tests to check if this happens correctly.
Add `x.py setup`
Closes#76503.
- Suggest `x.py setup` if config.toml doesn't exist yet
- Prompt for a profile if not given on the command line
- Print the configuration that will be used
- Print helpful starting commands after setup
- Link to the dev-guide after finishing
Install std for non-host targets
It seems reasonable that when configuring various targets you'd expect all of them to get std installed, even if you're not building compiler toolchains for each of those.
cc #76990
r? @alexcrichton
Invalidate local LLVM cache less often
This avoids a download of LLVM after every rebase. The downside to this is that if we land some patch affecting LLVM built in CI that breaks this option, but that PR does not update the LLVM submodule, we'll likely not notice until the next update -- but this seems unlikely to happen in practice and I am not personally worried about it.
r? @alexcrichton
Add `--keep-stage-std` to `x.py` for keeping only standard library artifacts
Unlike `--keep-stage 0`, `--keep-stage-std 0` will allow the stage 0 compiler artifacts (i.e., stage1/bin/rustc) to be rebuilt if it has changed. This allows contributors to iterate on later stages of the compiler in tandem with the standard library without needing to to rebuild the entire compiler. I often run into this when working on const-checking, since I may need to add a feature gate or make a small tweak to the standard library.
- Suggest `x.py setup` if config.toml doesn't exist yet (twice, once
before and once after the build)
- Prompt for a profile if not given on the command line
- Print the configuration file that will be used
- Print helpful starting commands after setup
- Link to the dev-guide after finishing
- Note that distro maintainers will see the changelog warning
This keeps only the `std` artifacts compiled by the given stage, not the
compiler. This is useful when working on the latter stages of the
compiler in tandem with the standard library, since you don't have to
rebuild the *entire* compiler when the standard library changes.
Don't download/sync llvm-project submodule if download-ci-llvm is set
llvm-project takes > 1GB storage space and a long time to download.
It's better to not download it unless needed.
Don't dynamically link LLVM tools unless rustc is too
This PR initially tried to support link-shared on all of our target platforms (other than Windows), but ran into a number of difficulties:
* LLVM doesn't really support a shared link on macOS (llvm-config runs into problems with the version suffix)
* LLVM doesn't seem to support a shared link when cross-compiling (the libLLVM.so ends up empty and symbols are not found)
So, this PR has now been revised such that we don't attempt to dynamically link LLVM tools (even if that would, otherwise, be supported) on targets where LLVM is statically linked to rustc. Currently that's basically everything except for x86_64-unknown-linux-gnu (where we dynamically link to avoid rerunning ThinLTO in each stage).
Follow-up to #76708.
Fixes#76698.
Add a changelog for x.py and nag contributors until they read it
Add a changelog for x.py
- Add a changelog and instructions for updating it
- Use `changelog-seen` in `config.toml` and `VERSION` in bootstrap to determine whether the changelog has been read. There's no way to tie reading the changelog to updating the version, so unfortunately they still have to update `config.toml` manually. Actually reading the changelog is optional, anyone can set `changelog-seen = N` without reading (although it's not recommended).
- Nag people if they haven't read the x.py changelog
+ Print message twice to make sure it's seen
- Give different error messages depending on whether the version needs to be updated or added
Closes https://github.com/rust-lang/rust/issues/76617
r? `@Mark-Simulacrum`
Fix cross compiling dist/build invocations
I am uncertain why the first commit is not affecting CI. I suspect it's because we pass --disable-docs on most of our cross-compilation builders. The second commit doesn't affect CI because CI runs x.py dist, not x.py build.
Both commits are standalone; together they should resolve#76733. The first commit doesn't really fix that issue but rather just fixes cross-compiled x.py dist, resolving a bug introduced in #76549.
- Add a changelog and instructions for updating it
- Use `changelog-seen` in `config.toml` and `VERSION` in bootstrap to determine whether the changelog has been read
- Nag people if they haven't read the x.py changelog
+ Print message twice to make sure it's seen
- Give different error messages depending on whether the version needs to be updated or added
Add sample defaults for config.toml
- Allow including defaults in `src/bootstrap/defaults` using `profile = "..."`.
- Add default config files, with a README noting they're experimental and asking you to open an issue if you run into trouble. The config files have comments explaining why the defaults are set.
- Combine config files using the `merge` dependency.
This introduces a new dependency on `merge` that hasn't yet been vetted.
I want to improve the output when `include = "x"` isn't found:
```
thread 'main' panicked at 'fs::read_to_string(&file) failed with No such file or directory (os error 2) ("configuration file did not exist")', src/bootstrap/config.rs:522:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failed to run: /home/joshua/rustc/build/bootstrap/debug/bootstrap test tidy
Build completed unsuccessfully in 0:00:00
```
However that seems like it could be fixed in a follow-up.
Closes#76619
- Allow including defaults in `src/bootstrap/defaults` using `profile = "..."`
- Add default config files
- Combine config files using the merge dependency.
- Add comments to default config files
- Add a README asking to open an issue if the defaults are bad
- Give a loud error if trying to merge `.target`, since it's not
currently supported
- Use an exhaustive match
- Use `<none>` in config.toml.example to avoid confusion
- Fix bugs in `Merge` derives
Previously, it would completely ignore the profile defaults if there
were any settings in `config.toml`. I sent an email to the `merge` maintainer
asking them to make the behavior in this commit the default.
This introduces a new dependency on `merge` that hasn't yet been vetted.
I want to improve the output when `include = "x"` isn't found:
```
thread 'main' panicked at 'fs::read_to_string(&file) failed with No such file or directory (os error 2) ("configuration file did not exist")', src/bootstrap/config.rs:522:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failed to run: /home/joshua/rustc/build/bootstrap/debug/bootstrap test tidy
Build completed unsuccessfully in 0:00:00
```
However that seems like it could be fixed in a follow-up.
This requires that bootstrap is run from the same worktree as the sources it'll
build, but this is basically required for the build to work anyway. You can
still run it from a different directory, just that the files it builds must be
beside it.
This moves build triple discovery for rustbuild from bootstrap.py into a build
script, meaning it will "just work" if building rustbuild via Cargo rather than
Python.
Move the version number to a plaintext file
The Rust version number is currently embedded in bootstrap's source code, which makes it hard to update it automatically or access it outside of ./x.py (as you'd have to parse the source code).
This PR moves the version number to a standalone plaintext file, which makes accessing or updating it trivial.
r? @Mark-Simulacrum
Don't generate bootstrap usage unless it's needed
Previously, `x.py` would unconditionally run `x.py build` to get the
help message. After https://github.com/rust-lang/rust/issues/76165,
when checking the CI stage was moved into `Config`, that would cause an
assertion failure (but only only in CI!):
```
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `1`,
right: `2`', src/bootstrap/config.rs:619:49
```
This changes bootstrap to only generate a help message when it needs
to (when someone passes `--help`).
r? @Mark-Simulacrum
This should fix the CI failures in https://github.com/rust-lang/rust/pull/76797 and https://github.com/rust-lang/rust/pull/75991.
Previously we would have some platforms where LLVM was linked to rustc
statically, but to the LLVM tools dynamically. That meant we were distributing
two copies of LLVM: one as a separate dylib and one statically linked in to
librustc_driver.
The Rust version number is currently embedded in bootstrap's source
code, which makes it hard to update it automatically or access it
outside of ./x.py (as you'd have to parse the source code).
This commit moves the version number to a standalone plaintext file,
which makes accessing or updating it trivial.
This isn't an issue for most folks who use x.py dist, which will directly depend
on this. But for x.py build, if we don't properly set target here rustdoc will
not be built.
Currently, there is not a default-on step for generating a rustc for a given
target either, so we will fail to build a rustc as well.
