Change generate-copyright to generate HTML, with cargo dependencies included
`x.py run generate-copyright` now produces `build/COPYRIGHT.html`. This includes a new format for in-tree dependencies, and also adds out-of-tree cargo dependencies.
After consulting expert opinion, I have elected to include every top-level:
* `*NOTICE*`
* `*AUTHOR*`
* `*LICENSE*`
* `*LICENCE*`, and
* `*COPYRIGHT*` file I can find - case-insensitive.
This is because the cargo package metadata's `author` field is not a list of copyright holders and does not meet the requirements of the Apache-2.0 license (which says you must include a NOTICE file with the binary if one was supplied by the author) nor the MIT license (which says you must include 'the above copyright notice').
I believe it would be appropriate to include this file with every Rust release, in order to do an even better job of appropriately recognising the efforts of the authors of the first-party and third-party libraries we are using here.
The output includes something like 524 copies of the Apache-2.0 text because they are not all identical. I think I count about 50 different variations by shasum - some differ in whitespace, while some have the boilerplate block at the bottom erroneously modified (don't modify the copy in the license, modify the copy you paste into your own source code!). Running `gzip` on the HTML file largely makes this problem go away, and the average browser is far happier with a ~6 MiB HTML file than the average Markdown viewer is with a ~6 MiB markdown file. But, if someone wants to, do they could submit a follow-up which de-dups the license text files and adds back-links to earlier identical copies (for some value of 'identical copy').
```console
$ xpy run generate-copyright
$ cd build
$ gzip -c COPYRIGHT.html > COPYRIGHT.gz
$ xz -c COPYRIGHT.html > COPYRIGHT.xz
$ ls -lh COPYRIGHT.*
-rw-r--r-- 1 jonathan staff 241K 29 Jul 17:19 COPYRIGHT.gz
-rw-r--r--@ 1 jonathan staff 6.6M 29 Jul 11:30 COPYRIGHT.html
-rw-r--r-- 1 jonathan staff 59K 29 Jul 17:19 COPYRIGHT.xz
```
Here's an example [COPYRIGHT.gz](https://github.com/user-attachments/files/16416147/COPYRIGHT.gz).
Remove the root Cargo.lock from the rust-src component
The Cargo.lock in library/ should be used instead. Including the Cargo.lock for the root workspace is both unnecessary and confusing.
Missed in https://github.com/rust-lang/rust/pull/128534
This tool now scans for cargo dependencies and includes any important looking license files.
We do this because cargo package metadata is not sufficient - the Apache-2.0 license says you have to include any NOTICE file, for example. And authors != copyright holders (cargo has the former, we must include the latter).
Move the standard library to a separate workspace
This ensures that the Cargo.lock packaged for it in the rust-src component is up-to-date, allowing rust-analyzer to run cargo metadata on the standard library even when the rust-src component is stored in a read-only location as is necessary for loading crates.io dependencies of the standard library.
This also simplifies tidy's license check for runtime dependencies as it can now look at all entries in library/Cargo.lock without having to filter for just the dependencies of runtime crates. In addition this allows removing an exception in check_runtime_license_exceptions that was necessary due to the compiler enabling a feature on the object crate which pulls in a dependency not allowed for the standard library.
While cargo workspaces normally enable dependencies of multiple targets to be reused, for the standard library we do not want this reusing to prevent conflicts between dependencies of the sysroot and of tools that are built using this sysroot. For this reason we already use an unstable cargo feature to ensure that any dependencies which would otherwise be shared get a different -Cmetadata argument as well as using separate build dirs.
This doesn't change the situation around vendoring. We already have several cargo workspaces that need to be vendored. Adding another one doesn't change much.
There are also no cargo profiles that are shared between the root workspace and the library workspace anyway, so it doesn't add any extra work when changing cargo profiles.
allow setting `link-shared` and `static-libstdcpp` with CI LLVM
These options also affect `compiler/rustc_llvm` builds. They should be configurable even when using CI LLVM.
r? ```@cuviper```
bootstrap: fix bug preventing the use of custom targets
the bug was caused by two factors:
1. only checking the RUST_TARGET_PATH form, not the full filepath form
2. indirectly trying to use the Debug presentation to get the file path
These options also affect `compiler/rustc_llvm` builds. They should be configurable
even when using CI LLVM.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
the bug was caused by two factors:
1. only checking the RUST_TARGET_PATH form, not the full filepath form
2. indirectly trying to use the Debug presentation to get the file path
This ensures that the Cargo.lock packaged for it in the rust-src
component is up-to-date, allowing rust-analyzer to run cargo metadata on
the standard library even when the rust-src component is stored in a
read-only location as is necessary for loading crates.io dependencies of
the standard library.
This also simplifies tidy's license check for runtime dependencies as it
can now look at all entries in library/Cargo.lock without having to
filter for just the dependencies of runtime crates. In addition this
allows removing an exception in check_runtime_license_exceptions that
was necessary due to the compiler enabling a feature on the object crate
which pulls in a dependency not allowed for the standard library.
While cargo workspaces normally enable dependencies of multiple targets
to be reused, for the standard library we do not want this reusing to
prevent conflicts between dependencies of the sysroot and of tools that
are built using this sysroot. For this reason we already use an unstable
cargo feature to ensure that any dependencies which would otherwise be
shared get a different -Cmetadata argument as well as using separate
build dirs.
This doesn't change the situation around vendoring. We already have
several cargo workspaces that need to be vendored. Adding another one
doesn't change much.
There are also no cargo profiles that are shared between the root
workspace and the library workspace anyway, so it doesn't add any extra
work when changing cargo profiles.
Update sysinfo version to 0.31.2
I needed to update `memchr` version (which was pinned in 36a16798f7). So let's see if it triggers the linker issue.
try-job: x86_64-mingw
improve bootstrap to allow selecting llvm tools individually
Everything works as before, + now bootstrap allows for individually selecting LLVM tools (e.g., `x dist opt llvm-dis`) to include in the dist artifact.
Everything works as before, + now bootstrap allows for individually selecting LLVM
tools (e.g., `x dist opt llvm-dis`) to include in the dist artifact.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
This allows the standard library to be compiled even with `download-rustc` enabled.
Which means it's no longer a requirement to compile `rustc` in order to compile `std`.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
`crates` field is handled in the `Step::make_run` just like in any other
`Std` implementation, so we don't need to resolve them in `Std::new`.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Since we now handle library crates properly, there's no need to panic for `no_std`
targets anymore.
`x doc library` now generates documentation for the `alloc` crate from standard library.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
This change unifies the `Step::run_make` logic and improves it by skipping
std specific crates for no_std targets.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Update the reference
This updates the reference to use the new mdbook-spec preprocessor, which is a Cargo library inside the reference submodule.
Note that this PR contains a bunch of bootstrap cleanup commits to assist with making sure the submodules are working correctly. All of the cleanup PRs should have a description in their commit. I'd be happy to move those to a separate PR if that makes review easier.
The main changes for the reference are:
- Move the `doc::Reference` bootstrap step out of the generic macro into a custom step.
- This step needs to build rustdoc because the new mdbook-spec plugin uses rustdoc for generating links.
- PATH is updated so that the rustdoc binary can be found.
- rustbook now includes the mdbook-spec plugin as a dependency.
- rustbook enables the mdbook-spec preprocessor.
I did a bunch of testing with the various commands and setups, such as:
- `submodules=true` and `submodules=false`
- having all submodules deinitialized
- not in a git repository
However, there are probably thousands of different permutations of different commands, settings, and environments, so there is a chance I'm missing something.
Bootstrap command refactoring: make command output API more bulletproof (step 7)
Continuation of https://github.com/rust-lang/rust/pull/127680.
This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller.
This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr).
Tracking issue: https://github.com/rust-lang/rust/issues/126819
r? `@onur-ozkan`
try-job: x86_64-msvc
I misread this one. It is only checking if LLVM needs to be rebuilt.
There is code below that handles the case where it is unable to compute
the stamp if the source is missing.
These are required 100% of the time, but they are almost always required
for any command that runs Cargo in the main workspace.
Ideally, initializing these two standard library submodules would be
lazy and only initialized when required (see
https://github.com/rust-lang/rust/pull/82653). However, it would require
updating these in almost every Step (anything that runs `cargo` in the
main workspace).
