For now this keeps all the configuration identical (AFAICT) but we'll
likely want to play with the specifics to move some of the slower
builders to larger machines and the faster builders to smaller machines,
likely reducing overall usage and improving CI times.
This removes a dependency on ntapi 0.3.x which failed to compiled with
latest beta.
rust-lang/rustc-perf@93dc60d995 removed
that dependency in the upstream rustc-perf.
Add more license annotations
This PR updates the `.reuse/dep5` file to include more accurate licensing data for everything in the repository (*excluding* submodules and dependencies). Some decisions were made in this PR:
* The standard copyright attribution for files maintained by us is "The Rust Project Developers (see https://thanks.rust-lang.org)", to avoid having to maintain an in-tree `AUTHORS` file.
* For files that have specific licensing terms, we added the terms to the `.reuse/dep5` rather than adding SPDX comments in the files themselves.
* REUSE picks up any comment/text line with `Copyright` on it, so I had to sprinkle around `REUSE-IgnoreStart` and `REUSE-IgnoreEnd` comments.
The rendered `COPYRIGHT` file is available at https://gist.github.com/pietroalbini/efb81103f69596d39758114f3f6a8688.
r? `@pnkfelix`
Store metrics from `metrics.json` to CI PGO timer
With this change, we'll be able to easily see how long does it take to compile LLVM vs `rustc`.
r? ```@Mark-Simulacrum```
Update the minimum external LLVM to 14
With this change, we'll have stable support for LLVM 14 through 16 (pending release).
For reference, the previous increase to LLVM 13 was #100460.
Run the tools builder on all PRs
Previously, it would only run on changes to subtrees, submodules, or select directories. That made it so that changes to the compiler that broke tools would only be detected on a full bors merge. This makes it so the tools builder runs by default, making it easier to catch breaking changes to clippy (which was the most affected).
r? ``@Mark-Simulacrum`` cc ``@pietroalbini`` ``@flip1995`` ``@m-ou-se``
Add nicer output to PGO build timer
This PR modifies the timer used in the PGO build script to contain nicer, hierarchical output of the individual build steps. It's not trivial to test locally, so I'll fire up a dist build right away.
r? ``@Mark-Simulacrum``
Previously, it would only run on changes to subtrees, submodules, or select directories.
That made it so that changes to the compiler that broke tools would only be detected on a full bors merge.
This makes it so the tools builder runs by default, making it easier to catch breaking changes to clippy (which was the most effected).
Rollup of 6 pull requests
Successful merges:
- #106575 (Suggest `move` in nested closure when appropriate)
- #106805 (Suggest `{var:?}` when finding `{?:var}` in inline format strings)
- #107500 (Add tests to assert current behavior of large future sizes)
- #107598 (Fix benchmarks in library/core with black_box)
- #107602 (Parse and recover from type ascription in patterns)
- #107608 (Use triple rather than arch for fuchsia test-runner)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Enable Cargo's sparse protocol in CI
This enables the sparse protocol in CI in order to exercise and dogfood it. This is intended test the production server in a real-world situation.
Closes#107342
Port pgo.sh to Python
This PR ports the `pgo.sh` multi stage build file from bash to Python, to make it easier to add new functionality and gather statistics. Main changes:
1) `pgo.sh` rewritten from Bash to Python. Jump from ~200 Bash LOC to ~650 Python LOC. Bash is, unsurprisingly, more concise for running scripts and binaries.
2) Better logging. Each separate stage is now clearly separated in logs, and the logs can be quickly grepped to find out which stage has completed or failed, and how long it took.
3) Better statistics. At the end of the run, there is now a table that shows the duration of the individual stages, along with a percentual ratio of the total workflow run:
```
2023-01-15T18:13:49.9896916Z stage-build INFO: Timer results
2023-01-15T18:13:49.9902185Z ---------------------------------------------------------
2023-01-15T18:13:49.9902605Z Build rustc (LLVM PGO): 1815.67s (21.47%)
2023-01-15T18:13:49.9902949Z Gather profiles (LLVM PGO): 418.73s ( 4.95%)
2023-01-15T18:13:49.9903269Z Build rustc (rustc PGO): 584.46s ( 6.91%)
2023-01-15T18:13:49.9903835Z Gather profiles (rustc PGO): 806.32s ( 9.53%)
2023-01-15T18:13:49.9904154Z Build rustc (LLVM BOLT): 1662.92s (19.66%)
2023-01-15T18:13:49.9904464Z Gather profiles (LLVM BOLT): 715.18s ( 8.46%)
2023-01-15T18:13:49.9914463Z Final build: 2454.00s (29.02%)
2023-01-15T18:13:49.9914798Z Total duration: 8457.27s
2023-01-15T18:13:49.9915305Z ---------------------------------------------------------
```
A sample run can be seen [here](https://github.com/rust-lang/rust/actions/runs/3923980164/jobs/6707932029).
I tried to keep the code compatible with Python 3.6 and don't use dependencies, which required me to reimplement some small pieces of functionality (like formatting bytes). I suppose that it shouldn't be so hard to upgrade to a newer Python or install dependencies in the CI container, but I'd like to avoid it if it won't be needed.
The code is in a single file `stage-build.py`, so it's a bit cluttered. I can also separate it into multiple files, although having it in a single file has some benefits. The code could definitely be nicer, but I'm a bit wary of introducing a lot of abstraction and similar stuff, as long as the code is stuffed into a single file.
Currently, the Python pipeline should faithfully mirror the bash pipeline one by one. After this PR, I'd like to try to optimize it, e.g. by caching the LLVM builds on S3.
r? `@Mark-Simulacrum`
Rollup of 11 pull requests
Successful merges:
- #96763 (Fix maintainer validation message)
- #106540 (Insert whitespace to avoid ident concatenation in suggestion)
- #106763 (print why a test was ignored if its the only test specified)
- #106769 (libtest: Print why a test was ignored if it's the only test specified.)
- #106798 (Implement `signum` with `Ord`)
- #107006 (Output tree representation on thir-tree)
- #107078 (Update wording of invalid_doc_attributes docs.)
- #107169 (Pass `--locked` to the x test tidy call)
- #107431 (docs: remove colon from time header)
- #107432 (rustdoc: remove unused class `has-srclink`)
- #107448 (When stamp doesn't exist, should say Error, and print path to stamp file)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Fix maintainer validation message
State clearly why maintainer verification is not activating, ruling out the user thinking they misconfigured something.
