Port pgo.sh to Python
This PR ports the `pgo.sh` multi stage build file from bash to Python, to make it easier to add new functionality and gather statistics. Main changes:
1) `pgo.sh` rewritten from Bash to Python. Jump from ~200 Bash LOC to ~650 Python LOC. Bash is, unsurprisingly, more concise for running scripts and binaries.
2) Better logging. Each separate stage is now clearly separated in logs, and the logs can be quickly grepped to find out which stage has completed or failed, and how long it took.
3) Better statistics. At the end of the run, there is now a table that shows the duration of the individual stages, along with a percentual ratio of the total workflow run:
```
2023-01-15T18:13:49.9896916Z stage-build INFO: Timer results
2023-01-15T18:13:49.9902185Z ---------------------------------------------------------
2023-01-15T18:13:49.9902605Z Build rustc (LLVM PGO): 1815.67s (21.47%)
2023-01-15T18:13:49.9902949Z Gather profiles (LLVM PGO): 418.73s ( 4.95%)
2023-01-15T18:13:49.9903269Z Build rustc (rustc PGO): 584.46s ( 6.91%)
2023-01-15T18:13:49.9903835Z Gather profiles (rustc PGO): 806.32s ( 9.53%)
2023-01-15T18:13:49.9904154Z Build rustc (LLVM BOLT): 1662.92s (19.66%)
2023-01-15T18:13:49.9904464Z Gather profiles (LLVM BOLT): 715.18s ( 8.46%)
2023-01-15T18:13:49.9914463Z Final build: 2454.00s (29.02%)
2023-01-15T18:13:49.9914798Z Total duration: 8457.27s
2023-01-15T18:13:49.9915305Z ---------------------------------------------------------
```
A sample run can be seen [here](https://github.com/rust-lang/rust/actions/runs/3923980164/jobs/6707932029).
I tried to keep the code compatible with Python 3.6 and don't use dependencies, which required me to reimplement some small pieces of functionality (like formatting bytes). I suppose that it shouldn't be so hard to upgrade to a newer Python or install dependencies in the CI container, but I'd like to avoid it if it won't be needed.
The code is in a single file `stage-build.py`, so it's a bit cluttered. I can also separate it into multiple files, although having it in a single file has some benefits. The code could definitely be nicer, but I'm a bit wary of introducing a lot of abstraction and similar stuff, as long as the code is stuffed into a single file.
Currently, the Python pipeline should faithfully mirror the bash pipeline one by one. After this PR, I'd like to try to optimize it, e.g. by caching the LLVM builds on S3.
r? `@Mark-Simulacrum`
Rollup of 11 pull requests
Successful merges:
- #96763 (Fix maintainer validation message)
- #106540 (Insert whitespace to avoid ident concatenation in suggestion)
- #106763 (print why a test was ignored if its the only test specified)
- #106769 (libtest: Print why a test was ignored if it's the only test specified.)
- #106798 (Implement `signum` with `Ord`)
- #107006 (Output tree representation on thir-tree)
- #107078 (Update wording of invalid_doc_attributes docs.)
- #107169 (Pass `--locked` to the x test tidy call)
- #107431 (docs: remove colon from time header)
- #107432 (rustdoc: remove unused class `has-srclink`)
- #107448 (When stamp doesn't exist, should say Error, and print path to stamp file)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Fix maintainer validation message
State clearly why maintainer verification is not activating, ruling out the user thinking they misconfigured something.
Rollup of 8 pull requests
Successful merges:
- #106618 (Disable `linux_ext` in wasm32 and fortanix rustdoc builds.)
- #107097 (Fix def-use dominance check)
- #107154 (library/std/sys_common: Define MIN_ALIGN for m68k-unknown-linux-gnu)
- #107397 (Gracefully exit if --keep-stage flag is used on a clean source tree)
- #107401 (remove the usize field from CandidateSource::AliasBound)
- #107413 (make more pleasant to read)
- #107422 (Also erase substs for new infcx in pin move error)
- #107425 (Check for missing space between fat arrow and range pattern)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Use stable metric for const eval limit instead of current terminator-based logic
This patch adds a `MirPass` that inserts a new MIR instruction `ConstEvalCounter` to any loops and function calls in the CFG. This instruction is used during Const Eval to count against the `const_eval_limit`, and emit the `StepLimitReached` error, replacing the current logic which uses Terminators only.
The new method of counting loops and function calls should be more stable across compiler versions (i.e., not cause crates that compiled successfully before, to no longer compile when changes to the MIR generation/optimization are made).
Also see: #103877
Remove HirId -> LocalDefId map from HIR.
Having this map in HIR prevents the creating of new definitions after HIR has been built.
Thankfully, we do not need it.
Based on https://github.com/rust-lang/rust/pull/103902
Remove `ControlFlow::{BREAK, CONTINUE}`
Libs-API decided to remove these in #102697.
Follow-up to #107023, which removed them from `compiler/`, but a couple new ones showed up since that was merged.
r? libs
Remove overlapping parts of multipart suggestions
This PR adds a debug assertion that the parts of a single substitution cannot overlap, fixes a overlapping substitution from the testsuite, and fixes https://github.com/rust-lang/rust/issues/106870.
Note that a single suggestion can still have multiple overlapping substitutions / possible edits, we just don't suggest overlapping replacements in a single edit anymore.
I've also included a fix for an unrelated bug where rustfix for `explicit_outlives_requirements` would produce multiple trailing commas for a where clause.
Compute generator saved locals on MIR
Generators are currently type-checked by introducing a `witness` type variable, which is unified with a `GeneratorWitness(captured types)` whose purpose is to ensure that the auto traits correctly migrate from the captured types to the `witness` type. This requires computing the captured types on HIR during type-checking, only to re-do it on MIR later.
This PR proposes to drop the HIR-based computation, and only keep the MIR one. This is done in 3 steps.
1. During type-checking, the `witness` type variable is never unified. This allows to stall all the obligations that depend on it until the end of type-checking. Then, the stalled obligations are marked as successful, and saved into the typeck results for later verification.
2. At type-checking writeback, `witness` is replaced by `GeneratorWitnessMIR(def_id, substs)`. From this point on, all trait selection involving `GeneratorWitnessMIR` will fetch the MIR-computed locals, similar to what opaque types do. There is no lifetime to be preserved here: we consider all the lifetimes appearing in this witness type to be higher-ranked.
3. After borrowck, the stashed obligations are verified against the actually computed types, in the `check_generator_obligations` query. If any obligation was wrongly marked as fulfilled in step 1, it should be reported here.
There are still many issues:
- ~I am not too happy having to filter out some locals from the checked bounds, I think this is MIR building that introduces raw pointers polluting the analysis;~ solved by a check specific to static variables.
- the diagnostics for captured types don't show where they are used/dropped;
- I do not attempt to support chalk.
cc `@eholk` `@jyn514` for the drop-tracking work
r? `@oli-obk` as you warned me of potential unsoundness
Revisit fix_is_ci_llvm_available logic
Fixes#107225
Now `supported_platforms` has a knowledge whether llvm asserts artifacts are available for particular host triple.
``@jyn514`` ``@albertlarsan68`` PTAL
Replace format flags u32 by enums and bools.
This gets rid of the `flags: u32` field where each bit has a special meaning, and replaces it by simple enums and booleans.
Part of #99012
rustdoc: Collect "rustdoc-reachable" items during early doc link resolution
This pass only needs to know about visibilities, attributes and reexports, so it can be run early, similarly to `compute_effective_visibilities` in rustc.
Results of this pass can be used to prune the list of extern impls early thus improving performance of https://github.com/rust-lang/rust/pull/94857.
rustdoc: use smarter encoding for playground URL
The old way would compress okay with DEFLATE, but this version makes uncompressed docs smaller, which matters for memory usage and stuff like `cargo doc`.
Try it out: <https://play.rust-lang.org/?code=fn+main()+{%0Alet+mut+v+=+Vec::new();%0Av.push(1+/+1);%0Aprintln!(%22{}%22,+v[0]);%0A}>
In local testing, this change shrinks sample pages by anywhere between 4.0% and 0.031%
$ du -b after.dir/std/vec/struct.Vec.html before.dir/std/vec/struct.Vec.html
759235 after.dir/std/vec/struct.Vec.html
781842 before.dir/std/vec/struct.Vec.html
100*((759235-781842)/781842)=-2.8
$ du -b after.dir/std/num/struct.Wrapping.html before.dir/std/num/struct.Wrapping.html
3194173 after.dir/std/num/struct.Wrapping.html
3204351 before.dir/std/num/struct.Wrapping.html
100*((3194173-3204351)/3204351)=-0.031
$ du -b after.dir/std/keyword.match.html before.dir/std/keyword.match.html
8151 after.dir/std/keyword.match.html
8495 before.dir/std/keyword.match.html
100*((8151-8495)/8495)=-4.0
Gzipped tarball sizes seem shrunk, but not by much.
du -s before.tar.gz after.tar.gz
69600 before.tar.gz
69480 after.tar.gz
100*((69480-69600)/69600)=-0.17
Sync rustc_codegen_cranelift
For cg_clif itself there have been a couple of bug fixes since the last sync, a Cranelift update and implemented all remaining simd platform intrinsics used by `std::simd`. (`std::arch` still misses a lot though) Most of the diff is from reworking of the cg_clif build system though.
r? `@ghost`
`@rustbot` label +A-codegen +A-cranelift +T-compiler
make `output_filenames` a real query
part of #105462
This may be a perf regression and is not obviously the right way forward. We may store this information in the resolver after freezing it for example.
The old way would compress okay with DEFLATE, but this version makes
uncompressed docs smaller, which matters for memory usage and stuff
like `cargo doc`.
Try it out: <https://play.rust-lang.org/?code=fn+main()+{%0Alet+mut+v+=+Vec::new();%0Av.push(1+/+1);%0Aprintln!(%22{}%22,+v[0]);%0A}>
In local testing, this change shrinks sample pages by anywhere between
4.0% and 0.031%
$ du -b after.dir/std/vec/struct.Vec.html before.dir/std/vec/struct.Vec.html
759235 after.dir/std/vec/struct.Vec.html
781842 before.dir/std/vec/struct.Vec.html
100*((759235-781842)/781842)=-2.8
$ du -b after.dir/std/num/struct.Wrapping.html before.dir/std/num/struct.Wrapping.html
3194173 after.dir/std/num/struct.Wrapping.html
3204351 before.dir/std/num/struct.Wrapping.html
100*((3194173-3204351)/3204351)=-0.031
$ du -b after.dir/std/keyword.match.html before.dir/std/keyword.match.html
8151 after.dir/std/keyword.match.html
8495 before.dir/std/keyword.match.html
100*((8151-8495)/8495)=-4.0
Gzipped tarball sizes seem shrunk, but not by much.
du -s before.tar.gz after.tar.gz
69600 before.tar.gz
69480 after.tar.gz
100*((69480-69600)/69600)=-0.17
