Switch to using the v2 resolver in most workspaces
Pinning the resolver to v1 was done in 5abff3753a ("Explicit set workspace.resolver ...") in order to suppress warnings. Since there is no specific reason not to use the new resolver and since it fixes issues, change to `resolver = "2"` everywhere except library.
Include a copy of `compiler-rt` source in the `download-ci-llvm` tarball
This will make it possible to experiment with allowing `download-ci-llvm` builds to build `library/profiler_builtins`, without needing to check out the `src/llvm-project` submodule.
By itself, this PR just adds the files to the tarball, but doesn't actually do anything with them. The idea is that once this is merged, it will then be much easier to proceed with work on the necessary bootstrap changes (using the real downloaded tarball), without having to rig up weird hacks to simulate downloading a modified tarball.
---
Adding these files to the compressed tarballs appears to increase its size by a negligible amount (<1 MB out of 400/800+ MB).
The uncompressed size is about 14 MB (out of 2+ GB for the whole tarball).
(The excluded test files would have been another 35 MB.)
With the new resolver, a few dependencies get brought in twice with
different licenses. For example, all dependencies from `wasm-tools`
gained Apache-2.0 and MIT options, and with the v2 resolver we were
using one version from before and one version from after this change.
This made tidy's license check difficult.
Update some minimum versions to remove duplicate dependencies and smooth
out license checking.
detect incompatible CI rustc options more precisely
Previously, the logic here was simply checking whether the option was set in `config.toml`. This approach was not manageable in our CI runners as we set so many options in config.toml. In reality, those values are not incompatible since they are usually the same value used to generate the CI rustc. Now, the new logic compares the configuration values with the values used to generate the CI rustc, so we get more precise results and make the process more manageable.
r? Kobzol
Blocker for https://github.com/rust-lang/rust/pull/122709
Previously, the logic here was simply checking whether the option was set in `config.toml`.
This approach was not manageable in our CI runners as we set so many options in config.toml.
In reality, those values are not incompatible since they are usually the same value used to generate
the CI rustc. Now, the new logic compares the configuration values with the values used to generate
the CI rustc, so we get more precise results and make the process more manageable.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Slightly refactor `TargetSelection` in bootstrap
Mostly a drive-by refactoring of `TargetSelection` to reduce some manual "windows-gnu" detection and also accesses to the `triple` field.
r? `@onur-ozkan`
bootstrap: don't use rustflags for `--rustc-args`
r? `@onur-ozkan`
This is going to require a bit of context.
https://github.com/rust-lang/rust/pull/47558 has added `--rustc-args` to `./x test` to allow passing flags when building `compiletest` tests. It was made specifically because using `RUSTFLAGS` would rebuild the compiler/stdlib, which would in turn require the flag you want to build tests with to successfully bootstrap.
#113178 made the request that it also works for other tests and doctests. This is not trivial to support across the board for `library`/`compiler` unit-tests/doctests and across stages. This issue was closed in #113948 by using `RUSTFLAGS`, seemingly incorrectly since https://github.com/rust-lang/rust/pull/123489 fixed that part to make it work.
Unfortunately #123489/#113948 have regressed the goals of `--rustc-args`:
- now we can't use rustc args that don't bootstrap, to run the UI tests: we can't test incomplete features. The new trait solver doesn't bootstrap, in-progress borrowck/polonius changes don't bootstrap, some other features are similarly incomplete, etc.
- using the flag now rebuilds everything from scratch: stage0 stdlib, stage1 compiler, stage1 stdlib. You don't need to re-do all this to compile UI tests, you only need the latter to run stdlib tests with a new flag, etc. This happens for contributors, but also on CI today. (Not to mention that in doing that it will rebuild things with flags that are not meant to be used, e.g. stdlib cfgs that don't exist in the compiler; or you could also imagine that this silently enables flags that were not meant to be enabled in this way).
Since then, https://github.com/rust-lang/rust/pull/125011/commits/bd71c71ea04b4a4f954e579c2a6d44113274846a has started using it to test a stdlib feature, relying on the fact that it now rebuilds everything. So #125011 also regressed CI times more than necessary because it rebuilds everything instead of just stage 1 stdlib.
It's not easy for me to know how to properly fix#113178 in bootstrap, but #113948/#123489 are not it since they regress the initial intent. I'd think bootstrap would have to know from the list of test targets that are passed that the `library` or `compiler` paths that are passed could require rebuilding these crates with different rustflags, probably also depending on stages. Therefore I would not be able to fix it, and will just try in this PR to unregress the situation to unblock the initial use-case.
It seems miri now also uses `./x miri --rustc-args` in this incorrect meaning to rebuild the `library` paths they support to run with the new args. I've not made any bootstrap changes related to `./x miri` in this PR, so `--rustc-args` wouldn't work there anymore. I'd assume this would need to use rustflags again but I don't know how to make that work properly in bootstrap, hence opening as draft, so you can tell me how to do that. I assume we don't want to break their use-case again now that it exists, even though there are ways to use `./x test` to do exactly that.
`RUSTFLAGS_NOT_BOOTSTRAP=flag ./x test library/std` is a way to run unit tests with a new flag without rebuilding everything, while with #123489 there is no way anymore to run tests with a flag that doesn't bootstrap.
---
edit: after review, this PR:
- renames `./x test --rustc-args` to `./x test --compiletest-rustc-args` as it only applies there, and cannot use rustflags for this purpose.
- fixes the regression that using these args rebuilt everything from scratch
- speeds up some CI jobs via the above point
- removes `./x miri --rustc-args` as only library tests are supported, needs to rebuild libstd, and `./x miri --compiletest-rustc-args` wouldn't work since compiletests are not supported.
bootstrap: clear miri ui-test deps when miri sysroot gets rebuilt
Second attempt after https://github.com/rust-lang/rust/pull/128683: seems like it's not the compiler changing that we care about, but the sysroot changing.
I did some local testing with sysroot rebuilds and it works fine for at least those cases I checked.
r? ``@onur-ozkan``
Slightly refactor `Flags` in bootstrap
The next step for https://github.com/rust-lang/rust/issues/126819 is to track commands executed inside `Config::parse`. This is quite challenging, because (tracked) command execution needs to access some state that is stored inside `Config`, which creates a sort of a chicken-and-egg problem.
I would like to first untangle `Config::parse` a little bit, which is what this PR starts with.
Tracking issue: https://github.com/rust-lang/rust/issues/126819
r? `@onur-ozkan`
Link `std` statically in `rustc_driver`
This makes `rustc_driver` statically link to `std`. This is done by not passing `-C prefer-dynamic` when building `rustc_driver`. However building `rustc-main` won't work currently as it tries to dynamically link to both `rustc_driver` and `std` resulting in a crate graph with `std` duplicated. To fix that new command line option `-Z prefer_deps_of_dynamic` is added which prevents linking to a dylib if there's a static variant of it already statically linked into another dylib dependency.
The main motivation for this change is to enable `#[global_allocator]` to be used in `rustc_driver` allowing overriding the allocator used in rustc on all platforms.
---
Instead of adding `-Z prefer_deps_of_dynamic`, this PR is changed to crate opt-in to the linking change via the `rustc_private` feature instead, as that would be typically needed to link to `rustc_driver` anyway.
---
try-job: aarch64-apple
try-job: x86_64-msvc
try-job: i686-mingw
try-job: dist-x86_64-msvc
try-job: aarch64-gnu
Promote aarch64-apple-darwin to Tier 1
This promotes aarch64-apple-darwin to Tier 1 status as per rust-lang/rfcs#3671 and tracking issue #73908. Not sure what else is necessary for this to impement the aforementioned RFC, however I figured I'd try. I did read in previous issues and PRs that the necessary infrastructure was already in place for the aarch64-apple-darwin target, and the RFC mentions the same. So this should be all thats necessary in order for the target to be promoted.
This is a recreation of my previous PR because I accidentally did an incorrect git rebase which caused unnecessary changes to various commit SHAs. So this PR is a recreation of my previous PR without said stumble. My bad.
Enable zstd for debug compression.
Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option.
See #120953
try-job: x86_64-gnu-tools
Disable verbose bootstrap command failure logging by default
One of my recent bootstrap command refactoring PRs enabled verbose logging of command failures by default. While this is great for debugging bootstrap, in many situations it's just too verbose and prevents the user from seeing the actual printed stdout/stderr, which usually contains much more useful information.
This PR reverts that logic, and only prints a detailed error when `-v` is passed to bootstrap.
r? ````@onur-ozkan````
add `builder-config` into tarball sources
This will be useful for certain scenarios where developers want to know how the tarball sources were generated. We also want this to check for CI rustc incompatible options on bootstrap.
Blocker for #122709
r? Kobzol
Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option.
Use static linking to avoid a new runtime dependency. Add an llvm.libzstd bootstrap option for LLVM
with zstd. Set it off by default except for the dist builder. Handle llvm-config --system-libs output
that contains static libraries.
