bootstrap: clear miri ui-test deps when miri sysroot gets rebuilt
Second attempt after https://github.com/rust-lang/rust/pull/128683: seems like it's not the compiler changing that we care about, but the sysroot changing.
I did some local testing with sysroot rebuilds and it works fine for at least those cases I checked.
r? ``@onur-ozkan``
Slightly refactor `Flags` in bootstrap
The next step for https://github.com/rust-lang/rust/issues/126819 is to track commands executed inside `Config::parse`. This is quite challenging, because (tracked) command execution needs to access some state that is stored inside `Config`, which creates a sort of a chicken-and-egg problem.
I would like to first untangle `Config::parse` a little bit, which is what this PR starts with.
Tracking issue: https://github.com/rust-lang/rust/issues/126819
r? `@onur-ozkan`
Link `std` statically in `rustc_driver`
This makes `rustc_driver` statically link to `std`. This is done by not passing `-C prefer-dynamic` when building `rustc_driver`. However building `rustc-main` won't work currently as it tries to dynamically link to both `rustc_driver` and `std` resulting in a crate graph with `std` duplicated. To fix that new command line option `-Z prefer_deps_of_dynamic` is added which prevents linking to a dylib if there's a static variant of it already statically linked into another dylib dependency.
The main motivation for this change is to enable `#[global_allocator]` to be used in `rustc_driver` allowing overriding the allocator used in rustc on all platforms.
---
Instead of adding `-Z prefer_deps_of_dynamic`, this PR is changed to crate opt-in to the linking change via the `rustc_private` feature instead, as that would be typically needed to link to `rustc_driver` anyway.
---
try-job: aarch64-apple
try-job: x86_64-msvc
try-job: i686-mingw
try-job: dist-x86_64-msvc
try-job: aarch64-gnu
Promote aarch64-apple-darwin to Tier 1
This promotes aarch64-apple-darwin to Tier 1 status as per rust-lang/rfcs#3671 and tracking issue #73908. Not sure what else is necessary for this to impement the aforementioned RFC, however I figured I'd try. I did read in previous issues and PRs that the necessary infrastructure was already in place for the aarch64-apple-darwin target, and the RFC mentions the same. So this should be all thats necessary in order for the target to be promoted.
This is a recreation of my previous PR because I accidentally did an incorrect git rebase which caused unnecessary changes to various commit SHAs. So this PR is a recreation of my previous PR without said stumble. My bad.
Enable zstd for debug compression.
Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option.
See #120953
try-job: x86_64-gnu-tools
Disable verbose bootstrap command failure logging by default
One of my recent bootstrap command refactoring PRs enabled verbose logging of command failures by default. While this is great for debugging bootstrap, in many situations it's just too verbose and prevents the user from seeing the actual printed stdout/stderr, which usually contains much more useful information.
This PR reverts that logic, and only prints a detailed error when `-v` is passed to bootstrap.
r? ````@onur-ozkan````
add `builder-config` into tarball sources
This will be useful for certain scenarios where developers want to know how the tarball sources were generated. We also want this to check for CI rustc incompatible options on bootstrap.
Blocker for #122709
r? Kobzol
Set LLVM_ENABLE_ZSTD alongside LLVM_ENABLE_ZLIB so that --compress-debug-sections=zstd is an option.
Use static linking to avoid a new runtime dependency. Add an llvm.libzstd bootstrap option for LLVM
with zstd. Set it off by default except for the dist builder. Handle llvm-config --system-libs output
that contains static libraries.
This will be useful for certain scenarios where developers want to know
how the tarball sources were generated. We also want this to check for CI
rustc incompatible options on bootstrap.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Change generate-copyright to generate HTML, with cargo dependencies included
`x.py run generate-copyright` now produces `build/COPYRIGHT.html`. This includes a new format for in-tree dependencies, and also adds out-of-tree cargo dependencies.
After consulting expert opinion, I have elected to include every top-level:
* `*NOTICE*`
* `*AUTHOR*`
* `*LICENSE*`
* `*LICENCE*`, and
* `*COPYRIGHT*` file I can find - case-insensitive.
This is because the cargo package metadata's `author` field is not a list of copyright holders and does not meet the requirements of the Apache-2.0 license (which says you must include a NOTICE file with the binary if one was supplied by the author) nor the MIT license (which says you must include 'the above copyright notice').
I believe it would be appropriate to include this file with every Rust release, in order to do an even better job of appropriately recognising the efforts of the authors of the first-party and third-party libraries we are using here.
The output includes something like 524 copies of the Apache-2.0 text because they are not all identical. I think I count about 50 different variations by shasum - some differ in whitespace, while some have the boilerplate block at the bottom erroneously modified (don't modify the copy in the license, modify the copy you paste into your own source code!). Running `gzip` on the HTML file largely makes this problem go away, and the average browser is far happier with a ~6 MiB HTML file than the average Markdown viewer is with a ~6 MiB markdown file. But, if someone wants to, do they could submit a follow-up which de-dups the license text files and adds back-links to earlier identical copies (for some value of 'identical copy').
```console
$ xpy run generate-copyright
$ cd build
$ gzip -c COPYRIGHT.html > COPYRIGHT.gz
$ xz -c COPYRIGHT.html > COPYRIGHT.xz
$ ls -lh COPYRIGHT.*
-rw-r--r-- 1 jonathan staff 241K 29 Jul 17:19 COPYRIGHT.gz
-rw-r--r--@ 1 jonathan staff 6.6M 29 Jul 11:30 COPYRIGHT.html
-rw-r--r-- 1 jonathan staff 59K 29 Jul 17:19 COPYRIGHT.xz
```
Here's an example [COPYRIGHT.gz](https://github.com/user-attachments/files/16416147/COPYRIGHT.gz).
Remove the root Cargo.lock from the rust-src component
The Cargo.lock in library/ should be used instead. Including the Cargo.lock for the root workspace is both unnecessary and confusing.
Missed in https://github.com/rust-lang/rust/pull/128534
This tool now scans for cargo dependencies and includes any important looking license files.
We do this because cargo package metadata is not sufficient - the Apache-2.0 license says you have to include any NOTICE file, for example. And authors != copyright holders (cargo has the former, we must include the latter).
Move the standard library to a separate workspace
This ensures that the Cargo.lock packaged for it in the rust-src component is up-to-date, allowing rust-analyzer to run cargo metadata on the standard library even when the rust-src component is stored in a read-only location as is necessary for loading crates.io dependencies of the standard library.
This also simplifies tidy's license check for runtime dependencies as it can now look at all entries in library/Cargo.lock without having to filter for just the dependencies of runtime crates. In addition this allows removing an exception in check_runtime_license_exceptions that was necessary due to the compiler enabling a feature on the object crate which pulls in a dependency not allowed for the standard library.
While cargo workspaces normally enable dependencies of multiple targets to be reused, for the standard library we do not want this reusing to prevent conflicts between dependencies of the sysroot and of tools that are built using this sysroot. For this reason we already use an unstable cargo feature to ensure that any dependencies which would otherwise be shared get a different -Cmetadata argument as well as using separate build dirs.
This doesn't change the situation around vendoring. We already have several cargo workspaces that need to be vendored. Adding another one doesn't change much.
There are also no cargo profiles that are shared between the root workspace and the library workspace anyway, so it doesn't add any extra work when changing cargo profiles.
allow setting `link-shared` and `static-libstdcpp` with CI LLVM
These options also affect `compiler/rustc_llvm` builds. They should be configurable even when using CI LLVM.
r? ```@cuviper```
bootstrap: fix bug preventing the use of custom targets
the bug was caused by two factors:
1. only checking the RUST_TARGET_PATH form, not the full filepath form
2. indirectly trying to use the Debug presentation to get the file path
These options also affect `compiler/rustc_llvm` builds. They should be configurable
even when using CI LLVM.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
the bug was caused by two factors:
1. only checking the RUST_TARGET_PATH form, not the full filepath form
2. indirectly trying to use the Debug presentation to get the file path
This ensures that the Cargo.lock packaged for it in the rust-src
component is up-to-date, allowing rust-analyzer to run cargo metadata on
the standard library even when the rust-src component is stored in a
read-only location as is necessary for loading crates.io dependencies of
the standard library.
This also simplifies tidy's license check for runtime dependencies as it
can now look at all entries in library/Cargo.lock without having to
filter for just the dependencies of runtime crates. In addition this
allows removing an exception in check_runtime_license_exceptions that
was necessary due to the compiler enabling a feature on the object crate
which pulls in a dependency not allowed for the standard library.
While cargo workspaces normally enable dependencies of multiple targets
to be reused, for the standard library we do not want this reusing to
prevent conflicts between dependencies of the sysroot and of tools that
are built using this sysroot. For this reason we already use an unstable
cargo feature to ensure that any dependencies which would otherwise be
shared get a different -Cmetadata argument as well as using separate
build dirs.
This doesn't change the situation around vendoring. We already have
several cargo workspaces that need to be vendored. Adding another one
doesn't change much.
There are also no cargo profiles that are shared between the root
workspace and the library workspace anyway, so it doesn't add any extra
work when changing cargo profiles.
Update sysinfo version to 0.31.2
I needed to update `memchr` version (which was pinned in 36a16798f7). So let's see if it triggers the linker issue.
try-job: x86_64-mingw
improve bootstrap to allow selecting llvm tools individually
Everything works as before, + now bootstrap allows for individually selecting LLVM tools (e.g., `x dist opt llvm-dis`) to include in the dist artifact.
Everything works as before, + now bootstrap allows for individually selecting LLVM
tools (e.g., `x dist opt llvm-dis`) to include in the dist artifact.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
This allows the standard library to be compiled even with `download-rustc` enabled.
Which means it's no longer a requirement to compile `rustc` in order to compile `std`.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
