CI: add a script for dynamically computing CI job matrix
It would be great if was easier to run specific CI workflows locally, and also to allow us to spawn a specific CI workflow by bors, to enable running arbitrary try builds. See discussion [here](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/CI.20workflows.20refactoring).
This PR is a first step in that direction.
- Moves the definition of CI runners and (for now) PR jobs into a separate `jobs.yml` file.
- Adds a simple Python script that reads the file, decides which jobs should be active for the current CI workflow, and prints them as JSON to their output.
- The PR job then reads this output and generates its job matrix based on it.
By moving the job definitions from `ci.yml` into a separate file, we can handle it programmatically, which should make it easier to both do local execution of CI jobs and also to do arbitrary try builds.
Errors started showing up, and I read somewhere that this might be
because of old ninja versions. This ninja version is indeed *ancient*.
```
multiple outputs aren't (yet?) supported by depslog; bring this up on the mailing list if it affects you
```
Improvements to building and CI for mingw/msys
I was getting error messages when trying to follow the build instructions the mingw build for Rust, and managed to track the issue down to an incomparability of Rust's bootstrap program with MSYS2's version of git. Essentially, the problem is that MSYS2's git works in emulated unix-y paths, but bootstrap expects a Windows path. I found a workaround for this by using relative paths instead of absolute paths.
Along with that fix, this PR also updates the build instructions for MinGW to be compatible with modern versions of MSYS2, and some changes to CI to make sure that MSYS2's version of git is tested. In particular, I'm suggesting using the [MSYS2 github action](https://github.com/marketplace/actions/setup-msys2) specially made for this purpose, which is much less hacky than the old approach and gives us more control of what packages are installed. I also cleaned up as many alternate versions of key tools as I could find from PATH, to avoid accidental usage, and cleaned up some abuses of the `CUSTOM_MINGW` environment variable.
This fixes https://github.com/rust-lang/rust/issues/105696 and fixes https://github.com/rust-lang/rust/issues/117567
Allowing the Xcode version to "float" based on whatever default GitHub
selects creates an unreliable environment. When GitHub changes the
default, we can have multiple jobs in the same run using different
versions as it rolls out across machines. It can also cause oscillation
between runs as different machines are used. It also causes
unpredictable timing when the updates happen.
This change helps ensure that the version that is used is pinned. The
downside is that it requires manually bumping the version, and the risk
that if we take too long, older Xcodes will be removed and that will
break the build.
Previously, it would only run on changes to subtrees, submodules, or select directories.
That made it so that changes to the compiler that broke tools would only be detected on a full bors merge.
This makes it so the tools builder runs by default, making it easier to catch breaking changes to clippy (which was the most effected).
See comment added for details on the test builder restriction. This is primarily
intended for macOS CI, but is likely to be a slight win on other builders too.
Re-enable submodule archive downloads.
This is effectively a revert of #98423 (though it keeps the `--depth 1` flag since that is still helpful).
GitHub has indicated that they have been working on the original issue, and my testing shows that the llvm-project archive download now succeeds 100% of the time.
This should save about a minute on every job.
The script wasn't referenced anywhere, and it's not useful anymore:
/opt/ghc is not present in new images, while /usr/share/dotnet is only
2.3 GB rather than 16 GB.
