Make sure to rebuild rustdoc if `src/rustdoc-json-types` is changed
I think `rustdoc-json-types` was more recently split out, so this download-rustc logic became outdated as it wasn't tracked. This PR adds `src/rustdoc-json-types` to be tracked for difference versus upstream, so that we properly rebuild rustdoc if it has changes versus upstream.
Fixesrust-lang/rust#142738.
### Local testing
This is not so easy to test locally because it requires download-rustc. To test this, you need to:
1. Disable `download-rustc` inhibition from bootstrap changes versus upstream, by including `:!src/bootstrap` in 255aa22082/src/bootstrap/src/core/config/config.rs (L67-L74).
2. Then, use a config like `profile = "tools"` which by default uses `download-rustc = "if-unchanged"`.
3. Run `./x test tests/rustdoc-json` one time, to "prime" initial build caches.
4. Change the `FORMAT_VERSION` in `src/rustdoc-json-types`, i.e.
```diff
diff --git a/src/rustdoc-json-types/lib.rs b/src/rustdoc-json-types/lib.rs
index 1f93895ae07..72a3720c7b4 100644
--- a/src/rustdoc-json-types/lib.rs
+++ b/src/rustdoc-json-types/lib.rs
``@@`` -38,7 +38,7 ``@@``
// are deliberately not in a doc comment, because they need not be in public docs.)
//
// Latest feature: Pretty printing of inline attributes changed
-pub const FORMAT_VERSION: u32 = 48;
+pub const FORMAT_VERSION: u32 = 666;
```
5. Observe that without this patch, `rustdoc-json` tests fail because `FORMAT_VERSION` mismatch. Observe that with this patch, rustdoc gets properly rebuilt and `rustdoc-json` tests pass.
cc ``@aDotInTheVoid``
r? Kobzol
Add config builder for bootstrap tests
I started writing a bunch of snapshot tests for build/check steps, and quickly realized that the current interface for defining them won't be enough, so I created a simple builder, which can scale to pretty much any kind of configuration in the future.
Use jemalloc for Clippy
The tool macros are annoying, we should IMO just get rid of them, create separate steps for each tool and (re)use some builders in them to share the build code.
r? `@ghost`
Assorted bootstrap cleanups (step 3)
I keep failing to unwrap the gordic knot of the logic of checking tools in bootstrap 😖 So in the meantime I at least want to upstream some cleanups I did along the way.
Since some time ago, we have separate steps for Clippy, so it shouldn't ever happen again that the check steps would be invoked with `builder.kind == Clippy`.
r? `@jieyouxu`
Clarify bootstrap tools description
The existence of `stage0-bootstrap-tools` suggests the possiblity of `stage1/N-bootstrap-tools`, but that's not really a thing. Also it doesn't fit the new bootstrap model, where `stageN` essentially means that it was built with a `stageN-1` compiler (except for std).
r? ``@jieyouxu``
Add `StepMetadata` to describe steps
This is used to replace the previous downcasting of executed steps, which wasn't very scalable. In addition to tests, we could also use the metadata e.g. for tracing.
r? ```@jieyouxu```
Actually take `--build` into account in bootstrap
I went back 20 *stable* versions of Rust and I couldn't find this flag actually being used. Despite some of our CI workflows actually set this flag (!).
I added destructuring of the flags to make sure that this doesn't happen again. It found one more duplicated CLI flag.
r? ```@jieyouxu```
Add spawn APIs for BootstrapCommand to support deferred command execution
This PR adds new deferred command support in the ExecutionContext and provides APIs to spawn commands and wait for their completion. This structure enables moving away from the start_process helper functions towards a more unified and reusable command execution flow.
r? ````@Kobzol````
Don't unwrap in enzyme builds in case of missing llvm-config
r? `@onur-ozkan`
For some reason x.py was now panicking in this location, so I also removed the unwrap here.
part 2 of https://github.com/rust-lang/rust/pull/140000, there shouldn't be other locations where we check for llvm-config.
Sized Hierarchy: Part I
This patch implements the non-const parts of rust-lang/rfcs#3729. It introduces two new traits to the standard library, `MetaSized` and `PointeeSized`. See the RFC for the rationale behind these traits and to discuss whether this change makes sense in the abstract.
These traits are unstable (as is their constness), so users cannot refer to them without opting-in to `feature(sized_hierarchy)`. These traits are not behind `cfg`s as this would make implementation unfeasible, there would simply be too many `cfg`s required to add the necessary bounds everywhere. So, like `Sized`, these traits are automatically implemented by the compiler.
RFC 3729 describes changes which are necessary to preserve backwards compatibility given the introduction of these traits, which are implemented and as follows:
- `?Sized` is rewritten as `MetaSized`
- `MetaSized` is added as a default supertrait for all traits w/out an explicit sizedness supertrait already.
There are no edition migrations implemented in this, as these are primarily required for the constness parts of the RFC and prior to stabilisation of this (and so will come in follow-up PRs alongside the const parts). All diagnostic output should remain the same (showing `?Sized` even if the compiler sees `MetaSized`) unless the `sized_hierarchy` feature is enabled.
Due to the use of unstable extern types in the standard library and rustc, some bounds in both projects have had to be relaxed already - this is unfortunate but unavoidable so that these extern types can continue to be used where they were before. Performing these relaxations in the standard library and rustc are desirable longer-term anyway, but some bounds are not as relaxed as they ideally would be due to the inability to relax `Deref::Target` (this will be investigated separately).
It is hoped that this is implemented such that it could be merged and these traits could exist "under the hood" without that being observable to the user (other than in any performance impact this has on the compiler, etc). Some details might leak through due to the standard library relaxations, but this has not been observed in test output.
**Notes:**
- Any commits starting with "upstream:" can be ignored, as these correspond to other upstream PRs that this is based on which have yet to be merged.
- This best reviewed commit-by-commit. I've attempted to make the implementation easy to follow and keep similar changes and test output updates together.
- Each commit has a short description describing its purpose.
- This patch is large but it's primarily in the test suite.
- I've worked on the performance of this patch and a few optimisations are implemented so that the performance impact is neutral-to-minor.
- `PointeeSized` is a different name from the RFC just to make it more obvious that it is different from `std::ptr::Pointee` but all the names are yet to be bikeshed anyway.
- `@nikomatsakis` has confirmed [that this can proceed as an experiment from the t-lang side](https://rust-lang.zulipchat.com/#narrow/channel/435869-project-goals/topic/SVE.20and.20SME.20on.20AArch64.20.28goals.23270.29/near/506196491)
- FCP in https://github.com/rust-lang/rust/pull/137944#issuecomment-2912207485Fixesrust-lang/rust#79409.
r? `@ghost` (I'll discuss this with relevant teams to find a reviewer)
This is used to replace the previous downcasting of executed steps, which wasn't very scalable. In addition to tests, we could also use the metadata e.g. for tracing.
I went back 20 *stable* versions of Rust and I couldn't find this flag actually being used. Despite some of our CI workflows actually set this flag (!).
Add initial version of snapshot tests to bootstrap
When making any changes to bootstrap (steps), it is very difficult to realize how does it affect various common bootstrap commands, and if everything still works as we expect it to. We are far away from having actual end-to-end tests, but what we could at least do is have a way of testing what steps does bootstrap execute in dry run mode. Now, we already have something like this in `src/bootstrap/src/core/builder/tests.rs`, however that is quite limited, because it only checks executed steps for a specific impl of `Step` and it does not consider step order.
Recently, when working on what I thought was one of the simplest possible step untanglings in bootstrap (https://github.com/rust-lang/rust/pull/142357), I ran into errors in tests that were quite hard to debug. Partly also because the current staging test diffs are multiline and use `Debug` output, so it's quite difficult for me to make sense of them.
In this PR, I introduce `insta`, which allows writing snapshot tests in a very simple way. With it, I want to allow writing tests that will clearly show us what is going on during bootstrap execution, and then write golden tests for `build/check/test` stage `0/1/2` for compiler/std/tools etc., to make sure that we don't regress something, and also to help with [#t-infra/bootstrap > Proposal to cleanup stages and steps after the redesign](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523488806), to help avoid a situation where we would (again) have to make a flurry of staging changes because of unexpected consequences.
In the snapshot tests, we currently render the build of rustc, std and LLVM. Currently I render the executed steps using downcasting, which is not super pretty, but it allows us to make the test rendering localized in one place, and it's IMO enough for now.
I implemented only a single test using the new machinery. Maybe if you take a look at it, you will understand why 😆 Bootstrap currently does some peculiar things, such as running a stage 0 std step (even though stage 0 std no longer exists) and running the Rustc stage 0 -> 1 step twice, once with a single crates, once with all rustc crates. So I think that even with this single step, there will be a bunch of things to fix in the near future...
The way we currently prepare the Config test fixtures is far from ideal, this is something I think ``@Shourya742`` could work on as a part of their GSoC project (remove as much command execution from Config construction as possible, actually run bootstrap on a temporary directory instead of running it on the rustc checkout, create a Builder-like API for creating the Config test fixtures).
r? ``@jieyouxu``
Assorted bootstrap cleanups (step 2)
Very small improvements designed towards making bootstrap tests less hacky/special, and towards making it possible to run bootstrap tests in parallel.
Best reviewed commit by commit.
r? ``@jieyouxu``
Add bootstrap option to compile a tool with features
Add an option to specify which features to build a tool with, e.g. it will be useful to build Miri with tracing enabled:
```toml
tool-config.miri.features = ["tracing"]
```
See [this Zulip thread](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Passing.20--features.20to.20Miri.20build.20using.20.2E.2Fx.2Epy/with/523564773) for the options considered. If the final decision will be different than what I wrote now, I will update the code as needed. The reason why the option is `tool-config.miri.features` instead of something like `tool-features.miri` is to possibly allow adding more tool-specific configurations in the future.
I didn't do any validation of the keys of the `tool-config` hashmap, since I saw that no validation is done on the `tools` hashset either.
I don't like much the fact that features can be chosen by various places of the codebase: `Step`s can have some fixed `extra_features`, `prepare_tool_cargo` will add features depending on some bootstrapping options, and the newly added option can also contribute features to tools. However I think it is out of scope of this PR to try to refactor all of that (if it even is refactorable), so I left a comment in the codebase explaining all of the sources of features I could find.
