Previously, it would only run on changes to subtrees, submodules, or select directories.
That made it so that changes to the compiler that broke tools would only be detected on a full bors merge.
This makes it so the tools builder runs by default, making it easier to catch breaking changes to clippy (which was the most effected).
See comment added for details on the test builder restriction. This is primarily
intended for macOS CI, but is likely to be a slight win on other builders too.
Re-enable submodule archive downloads.
This is effectively a revert of #98423 (though it keeps the `--depth 1` flag since that is still helpful).
GitHub has indicated that they have been working on the original issue, and my testing shows that the llvm-project archive download now succeeds 100% of the time.
This should save about a minute on every job.
The script wasn't referenced anywhere, and it's not useful anymore:
/opt/ghc is not present in new images, while /usr/share/dotnet is only
2.3 GB rather than 16 GB.
Add build metrics to rustbuild
This PR adds a new module of rustbuild, `ci_profiler`, whose job is to gather as much information as possible about the CI build as possible and store it in a JSON file uploaded to `ci-artifacts`. Right now for each step it collects:
* Type name and debug representation of the `Step` object.
* Duration of the step (excluding child steps).
* Systemwide CPU stats for the duration of the step (both single core and all cores).
* Which child steps were executed.
This is capable of replacing both the scripts to collect CPU stats and the `[TIMING]` lines in build logs (not yet removed, until we port our tooling to use the CI profiler). The format is also extensible to be able in the future to collect more information.
r? `@Mark-Simulacrum`
We currently upload approximately 166 GB/day into this bucket (estimate based on
duration of storage and total current size). My estimate is that this change
should decrease our costs (which are currently in credits) and is in the worst
case (if all objects are brought into hot storage due to unanticipated frequent
access) only going to add an additional ~$4 to the monthly bill. If access is
rare (as expected) to most objects then we expect to save approximately
~$350/month (after this change takes full effect in ~168 days).
Remove extra unshallow from cherry-pick checker
This is already done by 13db8440bb/src/ci/init_repo.sh (L32-L36) on the beta channel, and git throws an error if you attempt to unshallow an already non-shallow repository.
r? ```@pietroalbini```
This skips bumping Windows sccache because we run into compilation failures when
doing so (-m32 not supported by clang-cl?). Not clear on cause, but seems
easiest to just hold back.
This should avoid PGO-related failures encountered on Linux, and more broadly
seems like a good idea on other platforms as well (though it is likely not
necessary right this moment).
The only change here is a fix for `sys.platform` on Linux. Python 3.3
changed the API to return "linux" instead of "linux2"/"linux3", so this
commit uses `.startswith("python")` to make the code work on Python 3
without breaking Python 2.
Convert rustfmt from a submodule to a subtree
r? `@calebcartwright` cc `@Manishearth` `@Mark-Simulacrum`
The motivation is that submodule updates cause rustfmt to not be available on nightly a lot; most recently it was unavailable for over 10 days, causing the beta release to be delayed. Additionally this is much less work on the part of the rustfmt maintainers to keep the rustfmt compiling, since now people making breaking changes will be responsible for fixing them.
I kept the rustfmt git history so it looks like there are thousands of commits. The important commits are https://github.com/rust-lang/rust/compare/851dee3af9404bf399c3c4ffefe5105edb3debad~..pull/82208/head. This adds about 10 MB of git history, which is not terribly much compared to the 702 MB that already exist.
- Add `src/tools/rustfmt` to `x.py check`
- Fix CRLF issues with rustfmt tests (see commit for details)
- Use `rustc_private` instead of crates.io dependencies
This was already switched upstream and would have landed in the next submodule bump anyway. This just updates Cargo.lock for rust-lang/rust.
- Add `yansi-term` to the list of allowed dependencies.
This is a false positive - rustc doesn't actually use it, only rustfmt, but because it's activated by the cargo feature of a dependency, tidy gets confused. It's fairly innocuous in any case, it's used for color printing.
This would have happened in the next submodule bump.
- Remove rustfmt from the list of toolstate tools.
- Give a hard error if testing or building rustfmt fails.
- Update log to 0.4.14
This avoids a warning about semicolons in macros; see the commit for details.
- Don't add tools to the sysroot when they finish building.
This is the only change that could be considered a regression - this avoids a "colliding StableCrateId" error due to a bug in resolve (https://github.com/rust-lang/rust/issues/56935). The regression is that this rebuilds dependencies more often than strictly necessary. See the commit for details.
Fixes https://github.com/rust-lang/rust/issues/85226 (permanently). Closes https://github.com/rust-lang/rust/issues/82385. Helps with https://github.com/rust-lang/rust/issues/70651. Helps with https://github.com/rust-lang/rust/issues/80639.
Error out if a PR is sent to the wrong channel
It happened multiple times that a PR meant to go on beta ends up being opened (and occasionally merged) to master. This PR does two things:
* Moves the definition of the channel in `src/ci/channel` so it's easier for tools to read it. I was not sure whether to move it to `src/channel` (like `src/version`): ended up with `src/ci` as it's currently only used for CI, but I'm open to moving it to `src`. We'll need to update the release process after this.
* Adds a check on **non-bors** builds that errors out if the base branch is not the expected one for the currently defined channel. This will not cause problems for promotion PRs, as those PRs are meant to also update the channel name.
r? `@Mark-Simulacrum`
Retry clang+llvm download
We've been seeing a pretty high rate of spurious network failures (e.g., openssl
connection reset by peer). Not clear why, but let's add a retry.
r? `@pietroalbini`
