Fix artifact version/channel detection for stable
On stable, our artifacts are uploaded with the raw version number (e.g., 1.65.0), not the channel. This adjusts our detection logic to use the version number from src/version when we detect the stable channel.
This is really only important for stable channel re-builds, I think, but those do happen from time to time. I'm backporting a similar commit in https://github.com/rust-lang/rust/pull/103859 to make that PR pass CI.
On stable, our artifacts are uploaded with the raw version number (e.g.,
1.65.0), not the channel. This adjusts our detection logic to use the
version number from src/version when we detect the stable channel.
Detect unused files in `src/test/mir-opt` and error on them in tidy.
Closes#97564 .
Determining which files are generated by a given mir opt test is somewhat difficult. Because of this, we extract the logic for doing it out into a common crate that both compiletest and tidy can depend on. This avoids making compiletest a dependency of tidy which would negatively impact compile times for tidy.
Testing for this is that it catches 5 files that violated this lint (and removes them).
Enable RUSTC_BOOTSTRAP for a few steps
This forward-ports this commit so we don't need to keep applying it when branching beta (as done in 1.64, 1.65, and 1.66 beta bumps).
check lld version to choose correct option to disable multi-threading in tests
Testing compiler with 'use-lld = true' may be incorrect with old lld.
Flag, disabling multi-threading, should consider lld version.
r? ``@petrochenkov``
This reverts commit 3acb505ee5
(PR #101833).
The changes in this commit caused several bugs or at least
incompatibilies. For now we're reverting this commit and will re-land it
alongside fixes for those bugs.
compiletest: Refactor test rustcflags
Refactoring `host-rustcflags` and `target-rustcflags` from `Option<String>` to `Vec<String>`
Ref: #102438
r? `@Mark-Simulacrum`
Support excluding the generation of the standalone docs
For Ferrocene we need to exclude the generation of the standalone docs (which include the index page, which we want to replace with our own), but with the way bootstrap is currently implemented that proved not possible. This PR aims to support that.
The first problem is that the `doc::Standalone` step did two things: it generated the "standalone" documentation (which includes the index page and all the pages at the root of the documentation tree), but it also generated some files like `rust.css` and `version_info.html` that other step like `doc::TheBook` required. This meant generating the book required generating the index page, which made disabling the index page generation problematic.
The approach I took to fix the first problem is to split the step into `doc::Standalone` and `doc::SharedAssets`, with `doc::TheBook` now depending on `doc::SharedAssets`.
The second problem is that disabling the `doc::Standalone` proved to be tricky due to its path, `src/doc`. The path is accurate, as the source files for that step are `src/doc/*.md`. The problem is, bootstrap treats `--exclude` as a *suffix*, and so it also excluded the Cargo book whose source lives at `src/tools/cargo/src/doc`.
The approach I took to fix the second problem is to add the `standalone` path in addition to `src/doc`, so that you can pass `--exclude standalone`. I'm not fully happy with the solution, and the other idea I had was to just move the standalone docs source code to `src/doc/standalone`. I feel that second approach is cleaner, but also requires more changes and might require more consensus.
This PR is best reviewed commit-by-commit.
r? `@jyn514`
update Miri
I had to use a hacked version of josh to create this, so let's be careful with merging this and maybe wait a bit to see if the josh issue becomes more clear. But the history looks good to me, we are not adding duplicates of rustc commits that were previously mirrored to Miri.
Also I want to add some cross-testing of Miri in x.py.
All tools meant to be shipped with host toolchains only should be marked
as `ONLY_HOSTS = true`, but rust-analyzer was marked as `ONLY_HOSTS =
false` incorrectly. This meant that bootstrap attempted to build
rust-analyzer for cross-compilation-only targets, causing errors because
libstd is not present on some of them.
It will still be possible to cross-compile rust-analyzer by passing a
different --host flag to ./x, like you can cross-compile other tools.
tools/remote-test-{server,client}: Use /data/local/tmp on Android
The /data/tmp directory does not exist, at least not on recent versions of Android, which currently leads to test failures on that platform. I checked a virtual device running AOSP master and a Nexus 5 running Android Marshmallow and on both devices the /data/tmp directory does not exist and /data/local/tmp does, so let's switch to /data/local/tmp.
update to syn-1.0.102
This update removes the only `.gitignore` found in `rustc-src`:
vendor/syn/tests/.gitignore
vendor/syn-1.0.91/tests/.gitignore
vendor/syn-1.0.95/tests/.gitignore
To check-in `rustc-src` for hermetic builds in environments with
restrictive `.gitignore` policies, one has to remove these
`tests/.gitignore` and patch the respective
`.cargo-checksum.json`.`syn` >1.0.101 includes dtolnay/syn@3c49303bed,
which removes its `tests/.gitignore`. Now the `syn` crates.io package
has no `.gitignore`.
[`rustc-src`'s `vendor`][] is produced from the root `Cargo.toml`,
`src/tools/rust-analyzer/Cargo.toml`,
`compiler/rustc_codegen_cranelift/Cargo.toml`, and
`src/bootstrap/Cargo.toml`. `rustc_codegen_cranelift` does not use
`syn`.
[`rustc-src`'s `vendor`]:
c0784109da/src/bootstrap/dist.rs (L934-L940)
This was produced with:
cargo update --package syn --precise 1.0.102 \
cargo update --package syn --precise 1.0.102 \
--manifest-path src/tools/rust-analyzer/Cargo.toml
cargo update --package syn --precise 1.0.102 \
--manifest-path src/bootstrap/Cargo.toml
Use BOLT in CI to optimize LLVM
This PR adds an optimization step in the Linux `dist` CI pipeline that uses [BOLT](https://github.com/llvm/llvm-project/tree/main/bolt) to optimize the `libLLVM.so` library built by boostrap.
Steps:
- [x] Use LLVM 15 as a bootstrap compiler and use it to build BOLT
- [x] Compile LLVM with support for relocations (`-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-q"`)
- [x] Gather profile data using instrumented LLVM
- [x] Apply profile to LLVM that has already been PGOfied
- [x] Run with BOLT profiling on more benchmarks
- [x] Decide on the order of optimization (PGO -> BOLT?)
- [x] Decide how we should get `bolt` (currently we use the host `bolt`)
- [x] Clean up
The latest perf results can be found [here](https://github.com/rust-lang/rust/pull/94381#issuecomment-1258269440). The current CI build time with BOLT applied is around 1h 55 minutes.
Update CI to use Android NDK r25b
This commit updates the CI definitions to use the most recent Android LTS NDK release: r25b. Changes since the last NDK used by Rust negate the need to generate "standalone toolchains" and newer NDKs can be used in-place.
See https://developer.android.com/ndk/guides/other_build_systems#overview
Fix llvm-tblgen for cross compiling
- Let llvm-config tell us where to find its tools
- Add llvm-tblgen to rust-dev for cross-compiling
Fixes#86890.
r? ````@jyn514````
Make the `config.src` handling for downloadable bootstrap more conservative
In particular, this supports build directories within an unrelated git repository. Fixes https://github.com/rust-lang/rust/issues/102562.
As a side effect, it will fall back to the old logic when the source directory is being built from a tarball within an unrelated git repository. However, that second case is unsupported and untested; we reserve the right to break it in the future.
`@cr1901` can you confirm this fixes your problem?
cc `@kleisauke,` I believe this will also fix your issue (although your use case still isn't supported).
r? `@Mark-Simulacrum`
The /data/tmp directory does not exist, at least not on recent versions
of Android, which currently leads to test failures on that platform. I
checked a virtual device running AOSP master and a Nexus 5 running
Android Marshmallow and on both devices the /data/tmp directory does
not exist and /data/local/tmp does, so let's switch to /data/local/tmp.
Remove `-Ztime`
Because it has a lot of overlap with `-Ztime-passes` but is generally less useful. Plus some related cleanups.
Best reviewed one commit at a time.
r? `@davidtwco`
The compiler currently has `-Ztime` and `-Ztime-passes`. I've used
`-Ztime-passes` for years but only recently learned about `-Ztime`.
What's the difference? Let's look at the `-Zhelp` output:
```
-Z time=val -- measure time of rustc processes (default: no)
-Z time-passes=val -- measure time of each rustc pass (default: no)
```
The `-Ztime-passes` description is clear, but the `-Ztime` one is less so.
Sounds like it measures the time for the entire process?
No. The real difference is that `-Ztime-passes` prints out info about passes,
and `-Ztime` does the same, but only for a subset of those passes. More
specifically, there is a distinction in the profiling code between a "verbose
generic activity" and an "extra verbose generic activity". `-Ztime-passes`
prints both kinds, while `-Ztime` only prints the first one. (It took me
a close reading of the source code to determine this difference.)
In practice this distinction has low value. Perhaps in the past the "extra
verbose" output was more voluminous, but now that we only print stats for a
pass if it exceeds 5ms or alters the RSS, `-Ztime-passes` is less spammy. Also,
a lot of the "extra verbose" cases are for individual lint passes, and you need
to also use `-Zno-interleave-lints` to see those anyway.
Therefore, this commit removes `-Ztime` and the associated machinery. One thing
to note is that the existing "extra verbose" activities all have an extra
string argument, so the commit adds the ability to accept an extra argument to
the "verbose" activities.
Miri sync
This is a Miri sync created with my experimental fork of josh. We should probably not merge this yet, but we can use this to check if the sync looks the way it should.
r? `@oli-obk`
Before this commit, the path for the doc::Standalone step was "src/doc",
which is accurate as the standalone docs source files live at the root
of the "src/doc" directory tree.
Unfortunately, that caused bad interactions when trying to exclude it
with `--exclude src/doc`. When an exclusion is passed to bootstrap, it
will exclude all steps whose path *ends with* the exclusion, which
results in the Cargo book (src/tools/cargo/src/doc) to also be excluded.
To work around this problem, this commit adds the "standalone" path as
an alternate path for doc::Standalone, allowing `--exclude standalone`
to work without side effects.
