Dist build manifest
This PR makes two changes that should remove a significant chunk of the time spent in our release process: cloning the `rust-lang/rust` monorepo, all its submodules, and building `bootstrap` to then invoke `build-manifest`:
* `build-manifest` doesn't rely on a clone of the monorepo being present anymore. The only remaining bit of information it fetched from it (the Rust version) is instead bundled in the binary.
* A new "component" is added, `build-manifest`. That component includes a prebuilt version of the tool, and it's *not* included in the Rustup manifest. This will allow `promote-release` to directly invoke the tool without interacting with our build system.
* The Linux x86_64 CI is changed to also build the component mentioned above. It's the only CI builder tasked to do so, and to cleanly support this a new `--include-default-paths` flag was added to `./x.py`.
* The `BUILD_MANIFEST_NUM_THREADS` environment variable is added to configure the number of threads at runtime.
This PR is best reviewed commit-by-commit.
r? `@Mark-Simulacrum`
This commit changes the way build-manifest is invoked, to let it accept
the Rust version directly instead of requiring the path of the Rust
monorepo and letting build-manifest figure out the path on its own.
This allows to run build-manifest without a clone of the monorepo.
Always use the Rust version in package names
The format of the tarballs produced by CI is roughly the following:
{component}-{release}-{target}.{ext}
While on the beta and nightly channels `{release}` is just the channel name, on the stable channel is either the Rust version or the version of the component we're shipping:
cargo-0.47.0-x86_64-unknown-linux-gnu.tar.xz
clippy-0.0.212-x86_64-unknown-linux-gnu.tar.xz
llvm-tools-1.46.0-x86_64-unknown-linux-gnu.tar.xz
miri-0.1.0-x86_64-unknown-linux-gnu.tar.xz
rls-1.41.0-x86_64-unknown-linux-gnu.tar.xz
rust-1.46.0-x86_64-unknown-linux-gnu.tar.xz
...
This makes it really hard to get the package URL without having access to the manifest (and there is no manifest on ci-artifacts.rlo), as there is no consistent version number to use.
This PR addresses the problem by always using the Rust version number as `{release}` for the stable channel, regardless of the version number of the component we're shipping. I chose that instead of "stable" to avoid breaking the URL scheme *that* much.
Rustup should not be affected by this change, as it fetches the URLs from the manifest. Unfortunately we don't have a way to test other clients before making a stable release, as this change only affects the stable channel.
r? `@Mark-Simulacrum`
The format of the tarballs produced by CI is roughly the following:
{component}-{release}-{target}.{ext}
While on the beta and nightly channels `{release}` is just the channel
name, on the stable channel is either the Rust version or the version of
the component we're shipping:
cargo-0.47.0-x86_64-unknown-linux-gnu.tar.xz
clippy-0.0.212-x86_64-unknown-linux-gnu.tar.xz
llvm-tools-1.46.0-x86_64-unknown-linux-gnu.tar.xz
miri-0.1.0-x86_64-unknown-linux-gnu.tar.xz
rls-1.41.0-x86_64-unknown-linux-gnu.tar.xz
rust-1.46.0-x86_64-unknown-linux-gnu.tar.xz
...
This makes it really hard to get the package URL without having access
to the manifest (and there is no manifest on ci-artifacts.rlo), as there
is no consistent version number to use.
This commit addresses the problem by always using the Rust version
number as `{release}` for the stable channel, regardless of the version
number of the component we're shipping. I chose that instead of "stable"
to avoid breaking the URL scheme *that* much.
Rustup should not be affected by this change, as it fetches the URLs
from the manifest. Unfortunately we don't have a way to test other
clients before making a stable release, as this change only affects the
stable channel.
RunCompiler::new takes non-optional params, and optional
params can be set using set_*field_name* method.
finally `run` will forward all fields to `run_compiler`.
Fix LitKind's byte buffer to use refcounted slice
While working on adding a new lint for clippy (see https://github.com/rust-lang/rust-clippy/pull/6044) for avoiding shared ownership of "mutable buffer" types (such as using `Rc<Vec<T>>` instead of `Rc<[T]>`), I noticed a type exported from rustc_ast and used by clippy gets caught by the lint. This PR fixes the exported type.
This PR includes the actual change to clippy too, but I will open a PR directly against clippy for that part (although it will currently fail to build there).
This is a combination of 18 commits.
Commit #2:
Additional examples and some small improvements.
Commit #3:
fixed mir-opt non-mir extensions and spanview title elements
Corrected a fairly recent assumption in runtest.rs that all MIR dump
files end in .mir. (It was appending .mir to the graphviz .dot and
spanview .html file names when generating blessed output files. That
also left outdated files in the baseline alongside the files with the
incorrect names, which I've now removed.)
Updated spanview HTML title elements to match their content, replacing a
hardcoded and incorrect name that was left in accidentally when
originally submitted.
Commit #4:
added more test examples
also improved Makefiles with support for non-zero exit status and to
force validation of tests unless a specific test overrides it with a
specific comment.
Commit #5:
Fixed rare issues after testing on real-world crate
Commit #6:
Addressed PR feedback, and removed temporary -Zexperimental-coverage
-Zinstrument-coverage once again supports the latest capabilities of
LLVM instrprof coverage instrumentation.
Also fixed a bug in spanview.
Commit #7:
Fix closure handling, add tests for closures and inner items
And cleaned up other tests for consistency, and to make it more clear
where spans start/end by breaking up lines.
Commit #8:
renamed "typical" test results "expected"
Now that the `llvm-cov show` tests are improved to normally expect
matching actuals, and to allow individual tests to override that
expectation.
Commit #9:
test coverage of inline generic struct function
Commit #10:
Addressed review feedback
* Removed unnecessary Unreachable filter.
* Replaced a match wildcard with remining variants.
* Added more comments to help clarify the role of successors() in the
CFG traversal
Commit #11:
refactoring based on feedback
* refactored `fn coverage_spans()`.
* changed the way I expand an empty coverage span to improve performance
* fixed a typo that I had accidently left in, in visit.rs
Commit #12:
Optimized use of SourceMap and SourceFile
Commit #13:
Fixed a regression, and synched with upstream
Some generated test file names changed due to some new change upstream.
Commit #14:
Stripping out crate disambiguators from demangled names
These can vary depending on the test platform.
Commit #15:
Ignore llvm-cov show diff on test with generics, expand IO error message
Tests with generics produce llvm-cov show results with demangled names
that can include an unstable "crate disambiguator" (hex value). The
value changes when run in the Rust CI Windows environment. I added a sed
filter to strip them out (in a prior commit), but sed also appears to
fail in the same environment. Until I can figure out a workaround, I'm
just going to ignore this specific test result. I added a FIXME to
follow up later, but it's not that critical.
I also saw an error with Windows GNU, but the IO error did not
specify a path for the directory or file that triggered the error. I
updated the error messages to provide more info for next, time but also
noticed some other tests with similar steps did not fail. Looks
spurious.
Commit #16:
Modify rust-demangler to strip disambiguators by default
Commit #17:
Remove std::process::exit from coverage tests
Due to Issue #77553, programs that call std::process::exit() do not
generate coverage results on Windows MSVC.
Commit #18:
fix: test file paths exceeding Windows max path len
Replace `(Body, DefId)` with `Body` where possible
Follow-up to #77430.
I `grep`-ed for parameter lists in which a `Body` appeared within a few lines of a `DefId`, so it's possible that I missed some cases, but this should be pretty complete. Most of these changes were mechanical, but there's a few places where I started calling things "caller" and "callee" when multiple `DefId`s were in-scope at once. Also, we should probably have a helper function on `Body` that returns a `LocalDefId`. I can do that in this PR or in a follow-up.
Improve build-manifest to work with the improved promote-release
This PR makes some changes to build-manifest to have it work better with the other improvements I'm making to [promote-release](https://github.com/rust-lang/promote-release).
A new way to invoke the tool was added: `./x.py run src/tools/build-manifest`. The new invocation disables the generation of `.sha256` files and the generation of GPG signatures, as those steps are not tied to the Rust version we're building the manifest of: handling them in `promote-release` will improve the maintenability of our release process. Invocations through the old command (`./x.py dist hash-and-sign`) are referred inside the source code as "legacy". The new invocation also enables internal parallelism, disabled on legacy to avoid overloading our old server.
Improvements were also made on how the checksums included in the manifest are generated:
* The manifest is first generated with placeholder checksums, and then a function walks through the manifes and calculates only the needed hashes. Before this PR, all the hashes were calculated beforehand, including the hashes of unused files.
* Calculating the hashes is now done in parallel with rayon, to better utilize all the available disk bandwidth.
* The `sha2` crate is now used instead of the `sha256sum` CLI tool: this avoids the overhead of calling another process, but more importantly enables hardware acceleration whenever available (the `sha256sum` CLI tool doesn't support it at all).
r? @Mark-Simulacrum
This PR is best reviewed commit-by-commit.
Use `tracing` spans to trace the entire MIR interp stack
r? @RalfJung
While being very verbose, this allows really good tracking of what's going on. While I considered schemes like the previous indenter that we had (which we could get by using the `tracing-tree` crate), this will break down horribly with things like multithreaded rustc. Instead, we can now use `RUSTC_LOG` to restrict the things being traced. You could specify a filter in a way that only shows the logging of a specific frame.
If we lower the span's level to `debug`, then in `info` level logging we'd not see the frames, but in `debug` level we would see them. The filtering rules in `tracing` are super powerful, but I'm not sure if we can specify a filter so we do see `debug` level events, but *not* the `frame` spans. The documentation at https://docs.rs/tracing-subscriber/0.2.10/tracing_subscriber/struct.EnvFilter.html makes me think that we can only turn on things, not turn off things at a more precise level.
cc @hawkw
Write manifest for MAJOR.MINOR channel to enable rustup convenience
This connects to https://github.com/rust-lang/rustup/issues/794.
It's hard to remember if there have been patch releases for old versions
when you'd like to install the latest in a MAJOR.MINOR series.
When we're doing a stable release, we write duplicate manifests to
`stable`. With this change, only when we're doing a stable release, also
write duplicate manifests to `MAJOR.MINOR` to eventually enable rustup
(and any other tooling that builds Rust release URLs) to request, say,
`1.45` and get `1.45.2` (assuming `1.45.2` is the latest available
`1.45` and assuming that we never publish patch releases out of order).
I tested the best I could; it's a bit hard to get everything set up right
to be able to run the build-manifest tool. But I was able to run it with
a release of "1.45.2" and in addition to the files like `channel-rust-1.45.2.toml`
and `channel-rust-stable.toml` (and other manifests) that I got before this
change, I now get `channel-rust-1.45.toml`.
I believe this change to be safe to deploy as it does not change or remove
anything about manifests, just adds more. The actions in rust-central-station
that interact with manifests appear to use wildcards in such a way that it will
pick up these files without any problems.
There will need to be changes to `rustup` before `rustup install 1.45` will work,
but we can wait for a stable release and stable patch releases to happen with this
change before making the `rustup` changes, so that we're not committing to anything
before we know it works.
Rollup of 11 pull requests
Successful merges:
- #76851 (Fix 'FIXME' about using NonZeroU32 instead of u32.)
- #76979 (Improve std::sys::windows::compat)
- #77111 (Stabilize slice_ptr_range.)
- #77147 (Split sys_common::Mutex in StaticMutex and MovableMutex.)
- #77312 (Remove outdated line from `publish_toolstate` hook)
- #77362 (Fix is_absolute on WASI)
- #77375 (rustc_metadata: Do not forget to encode inherent impls for foreign types)
- #77385 (Improve the example for ptr::copy)
- #77389 (Fix some clippy lints)
- #77399 (BTreeMap: use Unique::from to avoid a cast where type information exists)
- #77429 (Link `new` method in `DefautHasher`s doc)
Failed merges:
r? `@ghost`
Remove outdated line from `publish_toolstate` hook
We no longer add `I-nominated` to toolstate failure issues since T-compiler changed its meeting preparation workflow.
This commit improves the way build-manifest calculates the checksums
included in the manifest, speeding it up:
* Instead of calculating all the hashes beforehand and then using the
ones we need, the manifest is first generated with placeholder hashes,
and then a function walks through the manifest and calculates only the
needed checksums.
* Calculating the checksums is now done in parallel with rayon, to
better utilize all the available disk bandwidth.
* Calculating the checksums now uses the sha2 crate instead of the
sha256sum CLI tool: this avoids the overhead of calling another
process, but more importantly uses hardware acceleration whenever
available (the CLI tool doesn't support it at all).
This connects to https://github.com/rust-lang/rustup/issues/794.
It's hard to remember if there have been patch releases for old versions
when you'd like to install the latest in a MAJOR.MINOR series.
When we're doing a stable release, we write duplicate manifests to
`stable`. With this change, only when we're doing a stable release, also
write duplicate manifests to `MAJOR.MINOR` to eventually enable rustup
(and any other tooling that builds Rust release URLs) to request, say,
`1.45` and get `1.45.2` (assuming `1.45.2` is the latest available
`1.45` and assuming that we never publish patch releases out of order).
