This extracts the linux-isms into variables, so that the script can be
extended to do PGO on windows. These variables will be overriden in a
few spots, in windows-specific blocks.
Remove unsupported options in configure.py
I've seen people using `optimize = false` and `full-bootstrap = true` in the past, without knowing
that they're not recommended. Remove `optimize` and a few other options that are always a bad idea,
and document that full-bootstrap is only for testing reproducible builds.
I've seen people using `optimize = false` and `full-bootstrap = true` in the past, without knowing
that they're not recommended. Remove `optimize` and a few other options that are always a bad idea,
and document that full-bootstrap is only for testing reproducible builds.
Clean up submodule checkout scripts
This is just some small cleanup:
* Removed unused CACHE_DIR stuff
* Removed duplicate fetch_github_commit_archive function which is no longer used
* Combined init_repo.sh and checkout-submodules.sh, as checkout-submodules.sh was doing nothing but calling init_repo.sh
The script wasn't referenced anywhere, and it's not useful anymore:
/opt/ghc is not present in new images, while /usr/share/dotnet is only
2.3 GB rather than 16 GB.
Add build metrics to rustbuild
This PR adds a new module of rustbuild, `ci_profiler`, whose job is to gather as much information as possible about the CI build as possible and store it in a JSON file uploaded to `ci-artifacts`. Right now for each step it collects:
* Type name and debug representation of the `Step` object.
* Duration of the step (excluding child steps).
* Systemwide CPU stats for the duration of the step (both single core and all cores).
* Which child steps were executed.
This is capable of replacing both the scripts to collect CPU stats and the `[TIMING]` lines in build logs (not yet removed, until we port our tooling to use the CI profiler). The format is also extensible to be able in the future to collect more information.
r? `@Mark-Simulacrum`
Add PID to LLVM PGO profile path
This is a continuation of https://github.com/rust-lang/rust/pull/97110, which adds PID to the filename pattern of LLVM profiles. It also adds some metrics to the pgo.sh script, so that we can observe how many profiles there are and how large are they.
r? `@lqd`
Move some settings DOM generation out of JS
The first commit reduce the JS size a bit by moving some DOM content generation into the HTML file directly.
The second commit is an update of the `browser-ui-test` version which improves `wait-for-*` command (if the element doesn't exist, it'll wait for it instead of failing).
r? ``@notriddle``
Switch CI bucket uploads to intelligent tiering
We currently upload approximately 166 GB/day into this bucket (estimate based on
duration of storage and total current size). My estimate is that this change
should decrease our costs (which are currently in credits) and is in the worst
case (if all objects are brought into hot storage due to unanticipated frequent
access) only going to add an additional ~$4 to the monthly bill. If access is
rare (as expected) to most objects then we expect to save approximately
~$350/month (after this change takes full effect in ~168 days).
r? ``@pietroalbini``
Add PID to PGO profile data filename
After experimenting with PGO, it looks like the generated profile data files can be sometimes overwritten if there is a race condition, because multiple `rustc` processes are usually invoked in parallel by `cargo`. Adding the PID to the resulting profile filename pattern makes sure that the profiles will be stored in separate files.
This generates ~20 GiB more space on disk on the CI run, but that seems harmless (?). Merging the profiles is not a bottleneck, the perf. run took the same amount of time as usually (~1h 24m).
r? `@lqd`
We currently upload approximately 166 GB/day into this bucket (estimate based on
duration of storage and total current size). My estimate is that this change
should decrease our costs (which are currently in credits) and is in the worst
case (if all objects are brought into hot storage due to unanticipated frequent
access) only going to add an additional ~$4 to the monthly bill. If access is
rare (as expected) to most objects then we expect to save approximately
~$350/month (after this change takes full effect in ~168 days).
