- Remove download-rustc handling from bootstrap.py
- Allow a custom `pattern` in `builder.unpack()`
- Only download rustc once another part of bootstrap depends on it.
This is somewhat necessary since the download functions rely on having a full
`Builder`, which isn't available until after config parsing finishes.
Make it possible to write doctests for bootstrap
This probably isn't super useful in practice, but it was easy to fix
and avoids confusing errors about mismatched versions between beta and the default toolchain.
This probably isn't super useful in practice, but it was easy to fix
and avoids confusing errors about mismatched versions between beta and the default toolchain.
Move `download-ci-llvm` out of bootstrap.py
This is ready for review. It has been tested on Windows, Linux, and NixOS.
The second commit ports the changes from https://github.com/rust-lang/rust/pull/95234 to Rust; I can remove it if desired.
Helps with https://github.com/rust-lang/rust/issues/94829.
As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that.
`@rustbot` label +A-rustbuild
Require all paths passed to `ShouldRun::paths` to exist on disk
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Builds on https://github.com/rust-lang/rust/pull/95901 and should not be merged before.
Make `x test --stage 2 compiler/rustc_XXX` faster to run
Previously, bootstrap unconditionally rebuilt the stage 2 compiler,
even if it had previously built stage 1. This changes it to reuse stage 1 if possible.
In particular, it no longer runs the following step:
```
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) -> x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu))
```
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Previously, bootstrap unconditionally rebuilt the stage 2 compiler,
even if it had previously built stage 1. This changes it to reuse stage 1 if possible.
Always use system `python3` on MacOS
This PR includes 2 changes:
1. Always use the system Python found at `/usr/bin/python3` on MacOS
2. Removes the hard requirement on having `python` in your system path if you didn't specify alternatives. The proposed change will instead attempt to find and use in order: `python` -> `python3` -> `python2`. This change isn't strictly necessary but without any change to this check, the original issue inspiring this change will still exist.
Fixes#95204
r? ```@jyn514```
Make it possible to run `cargo test` for bootstrap
Note that this only runs bootstrap's self-tests, not compiler or library tests.
Helps with https://github.com/rust-lang/rust/issues/94829.
Fix `x test src/librustdoc` with `download-rustc` enabled
The problem was two-fold:
- Bootstrap was hard-coding that unit tests should always run with stage1, not stage2, and
- It hard-coded the sysroot layout in stage1, which puts libLLVM.so in `lib/rustlib/` instead of just `lib/`.
This also takes the liberty of fixing `test src/librustdoc --no-doc`, which has been broken since it was first added. It would be nice at some point to unify this logic with other tests; I opened a Zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Inconsistency.20in.20.60x.20test.60
Fixes https://github.com/rust-lang/rust/issues/91071.
Previously, it would erroneously try to run the doc-tests anyway and give an error:
```
Doc-tests rustdoc
thread 'main' panicked at 'RUSTDOC_LIBDIR was not set', src/bootstrap/bin/rustdoc.rs:15:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: test failed, to rerun pass '--doc'
```
The problem was two-fold:
- Bootstrap was hard-coding that unit tests should always run with stage1, not stage2, and
- It hard-coded the sysroot layout in stage1, which puts libLLVM.so in `lib/rustlib/` instead of just `lib/`.
this also fixes a bug where bootstrap would try to use the fake `rustc` binary built by bootstrap -
cargo puts it in a different directory when using `cargo run` instead of x.py
The majority of the code is only used by either rustbuild or
rustc_llvm's build script. Rust_build is compiled once for rustbuild and
once for every stage. This means that the majority of the code in this
crate is needlessly compiled multiple times. By moving only the code
actually used by the respective crates to rustbuild and rustc_llvm's
build script, this needless duplicate compilation is avoided.
First, this reverts the `CFLAGS`/`CXXFLAGS` of #93918. Those flags are
already read by `cc` and populated into `Build` earlier on in the
process. We shouldn't be overriding that based on `CFLAGS`, since `cc`
also respects overrides like `CFLAGS_{TARGET}` and `HOST_CFLAGS`, which
we want to take into account.
Second, this adds the same capability to specify target-specific
versions of `LDFLAGS` as we have through `cc` for the `C*` flags:
https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables
Note that this also necessitated an update to compiletest to treat
CXXFLAGS separately from CFLAGS.
Building a dozen separate regexps for each test in compiletest consumes significant amounts of CPU cycles.
Using `RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER` stabilizes hashes calcuated for the individual tests so
no test-dependent normalization is needed. Hashes for the standard library still change so some
normalizations are still needed.
* Add wasm64 variants for inline assembly along the same lines as wasm32
* Update a few directives in libtest to check for `target_family`
instead of `target_arch`
* Update some rustc codegen and typechecks specialized for wasm32 to
also work for wasm64.
Don't rebuild GUI test crates every time you run test src/test/rustdoc-gui
This method has multiple advantages:
* It'll completely remove the rustdoc-GUI test doc folder if rustdoc was updated
* It'll rebuild GUI test crates only they have been updated
All in all, it's quite convenient! (even more with https://github.com/rust-lang/rust/pull/88816)
r? ```@Mark-Simulacrum```
Update all submodules that rustbuild doesn't depend on lazily
This only updates the submodules the first time they're needed, instead
of unconditionally the first time you run x.py.
Ideally, this would move *all* submodules to rustbuild and not exclude some tools and
backtrace. Unfortunately, cargo requires all `Cargo.toml` files in the
whole workspace to be present to build any crate.
On my machine, this takes the time for an initial submodule clone (for
`x.py --help`) from 55.70 to 15.87 seconds.
Helps with https://github.com/rust-lang/rust/issues/76653. Builds on https://github.com/rust-lang/rust/pull/86015 and should not be merged before (only the last commit is relevant).
This only updates the submodules the first time they're needed, instead
of unconditionally the first time you run x.py.
Ideally, this would move *all* submodules and not exclude some tools and
backtrace. Unfortunately, cargo requires all `Cargo.toml` files in the
whole workspace to be present to build any crate.
On my machine, this takes the time for an initial submodule clone (for
`x.py --help`) from 55.70 to 15.87 seconds.
This uses exactly the same logic as the LLVM update used, modulo some
minor cleanups:
- Use a local variable for `src.join(relative_path)`
- Remove unnecessary arrays for `book!` macro and make the macro simpler to use
- Add more comments
Allow to pass arguments to rustdoc-gui tool
Very convenient for testing. This is another part of https://github.com/rust-lang/rust/pull/86293
cc ``@jsha``
r? ``@Mark-Simulacrum``
Fix a bug in the linkchecker
There was a small typo in the linkchecker (in #85652) that caused it to report a `#` fragment link error pointing to the wrong file (it was displaying the path to the source file, not the target of the link).
This also includes a few other changes:
- Fixes the tests due to some changes in the redirect handling in #84703.
- Adds the tests to rustbuild to run whenever the linkchecker itself is run.
- Updates the tests to validate more of the output (so that a mistake like this would have been caught).
Closes#86144
The `RustdocGUI::should_run` condition spawns `npm list` several times
which adds up to seconds of wall-time.
Evaluate the condition lazily to to keep `./x.py test tidy` and similar
short-running tasks fast.
