Misc CI improvements
This PR contains some misc improvements to our CI configuration:
* The environment variables for MinGW builders were greatly simplified, with just `CUSTOM_MINGW=1` to tell the install scripts to install the vendored copy. All the others (`MINGW_URL`, `MINGW_DIR`, `MINGW_ARCHIVE` and `MSYS_BITS`) are detected either from the builder name or the environment.
* Collecting CPU stats and running the build were moved into scripts.
* Toolstate scripts validation was previously a separate step, ran just when `IMAGE=mingw-check`. This moves the validation code inside the actual image.
* Vendored copies are now fetched from https://ci-mirrors.rust-lang.org instead of directly from the bucket.
r? @alexcrichton
CentOS 5 only supports SSLv3 without SNI, and to get newer protocols
working we need to download and compile OpenSSL and cURL from our
mirror. Because of that, we can't use the CDN, as CloudFront requires
TLSv1 with SNI.
This commit changes the dist-x86_64-linux image to bypass the CDN for
OpenSSL and cURL.
This commit replaces the mirrors base URL contained in the MINGW_URL
with a CUSTOM_MINGW=1 environment variable. The mirrors base URL will be
fetched instead through the MIRRORS_BASE environment variable, defined
in src/ci/shared.sh.
Currently the `RUST_CONFIGURE_ARGS` variable apparently has a trailing
newline at the end of it due to the way it's configured in yaml. This
causes issues with MSVC's `install-clang.sh` step where the way the bash
syntax works out means that we drop the arg we're trying to add and it
doesn't actually get added!
The hopeful fix here is to tweak how we specify the yaml syntax to not
have a trailing newline, we'll see what CI says about this...
This updates the libc that the `wasm32-wasi` target links against to the
latest revision, mostly just bringing in minor bug fixes and minor wasm
size improvements.
ci: revert msys2 ca-certificates hack
The hack was added because upstream msys2 broke the ca-certificates package, but since then it has been fixed. This reverts CI to use the upstream package.
Part of #65767
Upload toolstates.json to rust-lang-ci2
This PR does two things:
* Following up with https://github.com/rust-lang/rust/pull/65202, it migrates deploying artifacts to CI in a script. Both uploading release artifacts and CPU stats were merged into the same script, designing it to be easily extended.
* Uploads the toolstate JSON to `rust-lang-ci2` along with the release artifacts, both for Linux and Windows. This is needed because @RalfJung wants to stop shipping MIRI when its tests are failing, and the toolstate repo doesn't have entries for each commit. Having the toolstate data (just for that specific commit) on `rust-lang-ci2` will simplify the code a lot.
r? @alexcrichton
cc @RalfJung
Before this commit toolstates.json was stored in /tmp and it wasn't
mounted outside the build container. That caused uploading the file in
the upload-artifacts task to fail, as the file was missing on the host.
Mounting /tmp/toolstates.json alone is not the best approach: if the
file is missing when the container is started the Docker engine will
create a *directory* named /tmp/toolstates.json.
The Docker issue could be solved by pre-creating an empty file named
/tmp/toolstates.json, but doing that could cause problems if bootstrap
fails to generate the file and the toolstate scripts receive an empty
JSON.
The approach I took in this commit is to instead mount a /tmp/toolstate
directory inside Docker, and create the toolstates.json file in it. That
also required a small bootstrap change to ensure the directory is
created if it's missing.
The hack was added because upstream msys2 broke the ca-certificates
package, but since then it has been fixed. This reverts CI to use the
upstream package.
Uploading the toolstate data for each commit will help our release
tooling understand which components are failing, to possibly skip
shipping broken tools to users.
We have a job in our CI (PR's x86_64-gnu-tools) that's supposed to run
only when a submodule is changed in the PR, and it works by having a
task at the start of the build that skips all the following tasks if the
condition isn't met.
Before this commit that task was gated with template parameters, which
is a unique feature of Azure Pipelines. To make our CI more generic this
commit switches the gate to use a simple environment variable plus a
condition, which should be supported on more CI providers.
