Since its inception rustbuild has always worked in three stages: one for
libstd, one for libtest, and one for rustc. These three stages were
architected around crates.io dependencies, where rustc wants to depend
on crates.io crates but said crates don't explicitly depend on libstd,
requiring a sysroot assembly step in the middle. This same logic was
applied for libtest where libtest wants to depend on crates.io crates
(`getopts`) but `getopts` didn't say that it depended on std, so it
needed `std` built ahead of time.
Lots of time has passed since the inception of rustbuild, however,
and we've since gotten to the point where even `std` itself is depending
on crates.io crates (albeit with some wonky configuration). This
commit applies the same logic to the two dependencies that the `test`
crate pulls in from crates.io, `getopts` and `unicode-width`. Over the
many years since rustbuild's inception `unicode-width` was the only
dependency picked up by the `test` crate, so the extra configuration
necessary to get crates building in this crate graph is unlikely to be
too much of a burden on developers.
After this patch it means that there are now only two build phasese of
rustbuild, one for libstd and one for rustc. The libtest/libproc_macro
build phase is all lumped into one now with `std`.
This was originally motivated by rust-lang/cargo#7216 where Cargo was
having to deal with synthesizing dependency edges but this commit makes
them explicit in this repository.
This patch supports less behavior than before, since specifiying stage 1
vs stage 2 is no longer possible, but that is presumably a somewhat rare
use case anyway, so not supporting it seems acceptable (and it can be
readded easily if desired).
Utilize -Zbinary-dep-depinfo in rustbuild
We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles.
Both of these are fixable in the longer term but this existing patch gives us the following benefits:
* We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes#54712.
* Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes#54008, fixes#50481.
* We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes#53284, and fixes#52719.
* Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes#49979 (but no reproduction steps in that issue). Fixes#59105.
cc #63012
ci: Attempt to skip a full rustc compile on dist*
Currently when we're preparing cross-compiled compilers it can take
quite some time because we have to build the compiler itself three
different times. The first is the normal bootstrap, the second is a
second build for the build platform, and the third is the actual target
architecture compiler. The second compiler was historically built
exclusively for procedural macros, and long ago we didn't actually need
it.
This commit tries out avoiding that second compiled compiler, meaning we
only compile rustc for the build platform only once. Some local testing
shows that this is promising, but bors is of course the ultimate test!
This extends a test in the previous commit to assert that we don't build
extra rustc compilers even when the "extended" option is set to true.
This involved some internal refactoring to have more judicious usage of
`compiler_for`, added in the previous commit, as well.
Various `dist::*` targets were refactored to be parameterized with a
`Compiler` instead of a `stage`/`host`, and then the various parameters
within the `Extended` target were tweaked to ensure that we don't ever
accidentally ask for a stage2 build compiler when we're distributing
something.
This commit furthers the previous one to ensure that we don't build an
extra stage of the compiler in CI. A test has been added to rustbuild to
ensure that this doesn't regress, and then in debugging this test it was
hunted down that the `dist::Std` target was the one erroneously pulling
in the wrong compiler.
The `dist::Std` step was updated to instead account for the "full
bootstrap" or not flag, ensuring that the correct compiler for compiling
the final standard library was used. This was another use of the
`force_use_stage1` function which was in theory supposed to be pretty
central, so existing users were all evaluated and a new function,
`Builder::compiler_for`, was introduced. All existing users of
`force_use_stage1` have been updated to use `compiler_for`, where the
semantics of `compiler_for` are similar to that of `compiler` except
that it doesn't guarantee the presence of a sysroot for the arguments
passed (as they may be modified).
Perhaps one day we can unify `compiler` and `compiler_for`, but the
usage of `Builder::compiler` is so ubiquitous it would take quite some
time to evaluate whether each one needs the sysroot or not, so it's
hoped that can be done in parallel.
Rework how bootstrap tools are built
This makes bootstrap tools buildable and testable in stage 0 with the downloaded bootstrap compiler, futhermore, it makes it such that they cannot be built in any other stage.
Notably, this will also mean that compiletest may need to wait a cycle before it can use changes to `libtest`, as it no longer depends on the in-tree libtest.
Introduce assembly tests suite
The change introduces a new test suite - **Assembly** tests. The motivation behind this is an ability to perform end-to-end codegen testing with LLVM backend. Turned out, NVPTX backend sometimes missing common Rust features (`i128` and libcalls in the past, and still full atomics support) due to different reasons.
Prior to this change, basic NVPTX assembly tests were implemented within `run-make` suite. Now, it's easier to write additional and maintain existing tests for the target.
cc @gnzlbg @peterhj
cc @eddyb I adjusted mangling scheme expectation, so there is no need to change the tests for #57967
Fixes rustdoc in stage 0, stage 1
When a request for rustdoc is passed for stage 0, x.py build --stage 0
src/tools/rustdoc or ensure(tool::Rustdoc { .. }) with top_stage = 0, we
return the rustdoc for that compiler (i.e., the beta rustdoc).
This fixes stage 0 of https://github.com/rust-lang/rust/issues/52186 as well as being part of general workflow improvements (making stage 0 testing for std work) for rustbuild.
The stage 1 fix (second commit) completely resolves the problem, so this fixes https://github.com/rust-lang/rust/issues/52186.
The RUSTDOC_LIBDIR should be rustc_libdir, not sysroot_libdir; rustdoc
is like the compiler and should link against rustc's libdir.
Some people currently (i.e., in general, may not be on master) have doc
tests working, but no attempt to determine why has been attempted.
