Utilize PGO for rustc linux dist builds
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater implementation time commitment,
implementing for platforms outside of x86_64-unknown-linux-gnu is skipped.
In practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour with this PR, up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
r? `@michaelwoerister` (but tell me if you don't want to / don't feel comfortable approving and we can find others)
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
`dsymutil` adds time to builds on Apple platforms for no clear benefit, and also
makes it more difficult for debuggers to find debug info. The compiler currently
defaults to running `dsymutil` to preserve its historical default, but when
compiling the compiler itself, we skip it by default since we know it's safe to
do so in that case.
Unfortunately, #80087 forgot to update the ci-llvm stamp, so the updated
ci-llvm tarball with `llvm-dwp` wasn't downloaded by users. This commit
updates the ci-llvm stamp to resolve that problem.
Signed-off-by: David Wood <david@davidtw.co>
This commit includes the `llvm-dwp` tool in the CI LLVM (which rustc
developers can download instead of building LLVM locally) - `llvm-dwp`
is required by Split DWARF which landed in PR #77117.
Signed-off-by: David Wood <david@davidtw.co>
This commit adds a Split DWARF compare mode to compiletest so that
debuginfo tests are also tested using Split DWARF in split mode (and
manually in single mode).
Signed-off-by: David Wood <david@davidtw.co>
`llvm-dwp` is required for linking the DWARF objects into DWARF packages
when using Split DWARF, especially given that rustc produces multiple
DWARF objects (one for each codegen unit).
Signed-off-by: David Wood <david@davidtw.co>
fix more clippy::complexity findings
fix clippy::unnecessary_filter_map
use if let Some(x) = .. instead of ...map(|x|) to conditionally run fns that return () (clippy::option_map_unit_fn)
fix clippy::{needless_bool, manual_unwrap_or}
don't clone types that are copy (clippy::clone_on_copy)
don't convert types into identical types with .into() (clippy::useless_conversion)
use strip_prefix over slicing (clippy::manual_strip)
r? ``@Dylan-DPC``
Fix rustup support in default_build_triple for python3
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
Follow-up to https://github.com/rust-lang/rust/pull/78513.
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
Validate lint docs separately.
This addresses some concerns raised in https://github.com/rust-lang/rust/pull/76549#issuecomment-727638552 about errors with the lint docs being confusing and cumbersome. Errors from validating the lint documentation were being generated during `x.py doc` (and `x.py dist`), since extraction and validation are being done in a single step. This changes it so that extraction and validation are separated, so that `x.py doc` will not error if there is a validation problem, and tests are moved to `x.py test src/tools/lint-docs`.
This includes the following changes:
* Separate validation to `x.py test`.
* Added some more documentation on how to more easily modify and test the docs.
* Added more help to the error messages to hopefully provide more information on how to fix things.
The first commit just moves the code around, so you may consider looking at the other commits for a smaller diff.
