Fix cross compiling dist/build invocations
I am uncertain why the first commit is not affecting CI. I suspect it's because we pass --disable-docs on most of our cross-compilation builders. The second commit doesn't affect CI because CI runs x.py dist, not x.py build.
Both commits are standalone; together they should resolve#76733. The first commit doesn't really fix that issue but rather just fixes cross-compiled x.py dist, resolving a bug introduced in #76549.
The Rust version number is currently embedded in bootstrap's source
code, which makes it hard to update it automatically or access it
outside of ./x.py (as you'd have to parse the source code).
This commit moves the version number to a standalone plaintext file,
which makes accessing or updating it trivial.
The directories for core, alloc, std, proc_macro, and test crates now
correspond directly to the crate name and stripping the "lib" prefix is
no longer necessary.
`rustc` allows passing in predefined target triples as well as JSON
target specification files. This change allows bootstrap to have the
first inkling about those differences. This allows building a
cross-compiler for an out-of-tree architecture (even though that
compiler won't work for other reasons).
Even if no one ever uses this functionality, I think the newtype
around the `Interned<String>` improves the readability of the code.
This should run much faster.
There are also some drive-by cleanups here to try to simplify things.
Also, the paths for in-tree crates are now displayed as relative
in `x.py test -h -v`.
On my machine, an error looks like:
Finished release [optimized] target(s) in 0.29s
Opening doc /git/rust/build/x86_64-unknown-linux-gnu/doc/std/index.html
command 'xdg-open (internal)' did not execute successfully; exit code: 4
command stderr:
gio: file:///git/rust/build/x86_64-unknown-linux-gnu/doc/std/index.html: Error when getting information for file “/git/rust/build/x86_64-unknown-linux-gnu/doc/std/index.html”: No such file or directory
Build completed successfully in 0:00:08
build-std compatible sanitizer support
### Motivation
When using `-Z sanitizer=*` feature it is essential that both user code and
standard library is instrumented. Otherwise the utility of sanitizer will be
limited, or its use will be impractical like in the case of memory sanitizer.
The recently introduced cargo feature build-std makes it possible to rebuild
standard library with arbitrary rustc flags. Unfortunately, those changes alone
do not make it easy to rebuild standard library with sanitizers, since runtimes
are dependencies of std that have to be build in specific environment,
generally not available outside rustbuild process. Additionally rebuilding them
requires presence of llvm-config and compiler-rt sources.
The goal of changes proposed here is to make it possible to avoid rebuilding
sanitizer runtimes when rebuilding the std, thus making it possible to
instrument standard library for use with sanitizer with simple, although
verbose command:
```
env CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS=-Zsanitizer=thread cargo test -Zbuild-std --target x86_64-unknown-linux-gnu
```
### Implementation
* Sanitizer runtimes are no long packed into crates. Instead, libraries build
from compiler-rt are used as is, after renaming them into `librusc_rt.*`.
* rustc obtains runtimes from target libdir for default sysroot, so that
they are not required in custom build sysroots created with build-std.
* The runtimes are only linked-in into executables to address issue #64629.
(in previous design it was hard to avoid linking runtimes into static
libraries produced by rustc as demonstrated by sanitizer-staticlib-link
test, which still passes despite changes made in #64780).
cc @kennytm, @japaric, @firstyear, @choller
This commit builds on #65501 continue to simplify the build system and
compiler now that we no longer have multiple LLVM backends to ship by
default. Here this switches the compiler back to what it once was long
long ago, which is linking LLVM directly to the compiler rather than
dynamically loading it at runtime. The `codegen-backends` directory of
the sysroot no longer exists and all relevant support in the build
system is removed. Note that `rustc` still supports a dynamically loaded
codegen backend as it did previously, it just no longer supports
dynamically loaded codegen backends in its own sysroot.
Additionally as part of this the `librustc_codegen_llvm` crate now once
again explicitly depends on all of its crates instead of implicitly
loading them through the sysroot. This involved filling out its
`Cargo.toml` and deleting all the now-unnecessary `extern crate`
annotations in the header of the crate. (this in turn required adding a
number of imports for names of macros too).
The end results of this change are:
* Rustbuild's build process for the compiler as all the "oh don't forget
the codegen backend" checks can be easily removed.
* Building `rustc_codegen_llvm` is much simpler since it's simply
another compiler crate.
* Managing the dependencies of `rustc_codegen_llvm` is much simpler since
it's "just another `Cargo.toml` to edit"
* The build process should be a smidge faster because there's more
parallelism in the main rustc build step rather than splitting
`librustc_codegen_llvm` out to its own step.
* The compiler is expected to be slightly faster by default because the
codegen backend does not need to be dynamically loaded.
* Disabling LLVM as part of rustbuild is still supported, supporting
multiple codegen backends is still supported, and dynamic loading of a
codegen backend is still supported.
This commit changes the return type of `Builder::cargo` to return a
builder that allows dynamically adding more `RUSTFLAGS` values
after-the-fact. While not used yet, this will later be used to delete
more of `rustc.rs`
Previously when building documentation for the standard library we'd
copy all the files 5 times, and these files include libcore/libstd docs
which are huge! This commit instead only copies the files after rustdoc
has been run for each crate, reducing the number of redundant copies
we're making.
Fix build failure in case file doesn't exist
It fixes the following issue:
```bash
$ ./x.py test src/tools/linkchecker ./build/x86_64-apple-darwin/doc/ --stage 1
Updating only changed submodules
Submodules updated in 0.05 seconds
Finished dev [unoptimized] target(s) in 0.15s
thread 'main' panicked at 'source "/Users/imperio/rust/rust/build/x86_64-apple-darwin/doc/version_info.html" failed to get metadata: No such file or directory (os error 2)', src/build_helper/lib.rs:179:19
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
failed to run: /Users/imperio/rust/rust/build/bootstrap/debug/bootstrap test src/tools/linkchecker ./build/x86_64-apple-darwin/doc/ --stage 1
Build completed unsuccessfully in 0:00:01
```
If the file doesn't exist, it makes sense anyway to just run the command in order to generate it.
r? @Mark-Simulacrum
Since its inception rustbuild has always worked in three stages: one for
libstd, one for libtest, and one for rustc. These three stages were
architected around crates.io dependencies, where rustc wants to depend
on crates.io crates but said crates don't explicitly depend on libstd,
requiring a sysroot assembly step in the middle. This same logic was
applied for libtest where libtest wants to depend on crates.io crates
(`getopts`) but `getopts` didn't say that it depended on std, so it
needed `std` built ahead of time.
Lots of time has passed since the inception of rustbuild, however,
and we've since gotten to the point where even `std` itself is depending
on crates.io crates (albeit with some wonky configuration). This
commit applies the same logic to the two dependencies that the `test`
crate pulls in from crates.io, `getopts` and `unicode-width`. Over the
many years since rustbuild's inception `unicode-width` was the only
dependency picked up by the `test` crate, so the extra configuration
necessary to get crates building in this crate graph is unlikely to be
too much of a burden on developers.
After this patch it means that there are now only two build phasese of
rustbuild, one for libstd and one for rustc. The libtest/libproc_macro
build phase is all lumped into one now with `std`.
This was originally motivated by rust-lang/cargo#7216 where Cargo was
having to deal with synthesizing dependency edges but this commit makes
them explicit in this repository.
This is no longer used by the index generator and was always an unstable
compiler detail, so strip it out.
This also leaves in RUSTC_ERROR_METADATA_DST since the stage0 compiler
still needs it to be set.
