Fix `download-ci-llvm` NixOS patching for `.so`s.
See https://github.com/rust-lang/rust/pull/95170#discussion_r872960686 - in short, `Path::ends_with` doesn't do the same thing as `str::ends_with`, and can only be used to check for whole file names, not extensions.
With this PR, I get the full suite of:
```
extracting /home/eddy/Projects/rust-A/build/cache/llvm-ebb80ec4e90f8622440f3e33562db0d6e6c66555-true/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz to /home/eddy/Projects/rust-A/build/x86_64-unknown-linux-gnu/ci-llvm
info: you seem to be using Nix. Attempting to patch /home/eddy/Projects/rust-A/build/x86_64-unknown-linux-gnu/ci-llvm/bin/llvm-config
/nix/store/r4bzq2xilvv8fmqjg626hzwi22ah3hf4-rust-stage0-dependencies
info: you seem to be using Nix. Attempting to patch /home/eddy/Projects/rust-A/build/x86_64-unknown-linux-gnu/ci-llvm/bin/FileCheck
info: you seem to be using Nix. Attempting to patch /home/eddy/Projects/rust-A/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so
```
(that `libLLVM-14-rust-1.62.0-nightly.so` at the end having been missing before)
r? `@Mark-Simulacrum` cc `@jyn514`
Add compiletest and bootstrap "--skip" option forwarded to libtest
With this PR, "x.py test --skip SKIP ..." will run the specified test suite, but forward "--skip SKIP" to the test tool. libtest already supports this option. The PR also adds it to compiletest which itself just forwards it to libtest.
Adds the functionality requested in https://github.com/rust-lang/rust/issues/96342. This is useful to work around tests broken upstream.
https://github.com/rust-lang/rust/issues/96362#issuecomment-1108609893 is the specific test issue my project is trying to work around.
Fix running bootstrap tests on a fresh clone
In #96303, I changed the tests not to manage submodules, with the main
goal of avoiding a clone for llvm-project. Unfortunately, there are some tests
which depend on submodules - I didn't notice locally because they were already checked out for me,
and CI doesn't use submodule handling at all. Fresh clones, however, were impacted:
```
failures:
---- builder::tests::defaults::doc_default stdout ----
thread 'main' panicked at 'fs::read_dir(builder.src.join(&relative_path).join("redirects")) failed with No such file or directory (os error 2)', src/bootstrap/doc.rs:232:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- builder::tests::dist::dist_only_cross_host stdout ----
thread 'main' panicked at 'fs::read_to_string(&toml_file_name) failed with No such file or directory (os error 2)', src/bootstrap/lib.rs:1314:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Try and get the best of both worlds by only checking out the submodules actually used in tests.
In #96303, I changed the tests not to manage submodules, with the main
goal of avoiding a clone for llvm-project. Unfortunately, there are some tests
which depend on submodules - I didn't notice locally because they were already checked out for me,
and CI doesn't use submodule handling at all. Fresh clones, however, were impacted:
```
failures:
---- builder::tests::defaults::doc_default stdout ----
thread 'main' panicked at 'fs::read_dir(builder.src.join(&relative_path).join("redirects")) failed with No such file or directory (os error 2)', src/bootstrap/doc.rs:232:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- builder::tests::dist::dist_only_cross_host stdout ----
thread 'main' panicked at 'fs::read_to_string(&toml_file_name) failed with No such file or directory (os error 2)', src/bootstrap/lib.rs:1314:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Try and get the best of both worlds by only checking out the submodules actually used in tests.
libtest already supports a "--skip SUBSTRING" arg which excludes any
test names matching SUBSTRING.
This adds a "--skip" argument to compiletest and bootstrap which is
forwarded to libtest.
Make "Assemble stage1 compiler" orders of magnitude faster
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
Improve error for missing cmake
This PR updates the error message for a missing `cmake` to be more in line with the error message for a missing installation of `ninja`.
The original issue, (#90679), suggests that both `ninja` and `cmake` are only needed for building LLVM, so I have included the suggestion from `ninja` to set `download-ci-llvm = true` if the user would rather download LLVM. If `cmake` actually is used in other areas, I can remove that part of the message.
Fixes: #90679
Rollup of 10 pull requests
Successful merges:
- #96336 (Link to correct `as_mut` in docs for `pointer::as_ref`)
- #96586 (Add aliases for std::fs::canonicalize)
- #96667 (Add regression test)
- #96671 (Remove hard links from `env::current_exe` security example)
- #96726 (Add regression and bug tests)
- #96756 (Enable compiler-docs by default for `compiler`, `codegen`, and `tools` profiles)
- #96757 (Don't constantly rebuild clippy on `x test src/tools/clippy`.)
- #96769 (Remove `adx_target_feature` feature from active features list)
- #96777 (Make the test `check-pass` not to produce a JSON file)
- #96822 (Enforce quote rule for JS source code)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Don't constantly rebuild clippy on `x test src/tools/clippy`.
This happened because the `SYSROOT` variable was set for `x test`, but not `x build`.
Set it consistently for both to avoid unnecessary rebuilds.
This is a very small step towards https://github.com/rust-lang/rust/issues/76495.
Enable compiler-docs by default for `compiler`, `codegen`, and `tools` profiles
I had this overridden locally for a while and realized just now it should probably just be a default.
Enable cfg checking of cargo features for everything but std
This PR enable `cfg` checking of cargo features for everything but std, it also adds a `FIXME` to myself.
> Note: `std`, `alloc` and `core` imports some dependencies by #[path] (like
> backtrace, core_simd, std_float, ...), those dependencies have their own features
> but cargo isn't involved in the #[path] and so cannot pass the complete list of
> features, so for that reason we don't enable checking of features for std.
r? `@Mark-Simulacrum`
bootstrap: bsd platform flags for split debuginfo
Addresses https://github.com/rust-lang/rust/pull/96597#issuecomment-1118905025.
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using split debuginfo - this commit provides it for all BSD targets.
We should probably work out a better way of handling the stability of the split debuginfo flag - all options for the flag are unstable but one of them is the default for each platform already.
cc `@m-ou-se`
r? `@Mark-Simulacrum`
[bootstrap] Give a better error when trying to run a path with no registered step
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using
split debuginfo - this commit provides it for all BSD targets.
Signed-off-by: David Wood <david.wood@huawei.com>
openbsd: unbreak build on native platform
after #95612, only linux and windows target are build with `-Zunstable-options`, but others platforms might use `-Csplit-debuginfo`
currently, without this PR, the build of rustc on OpenBSD fails with:
```
Building stage0 tool unstable-book-gen (x86_64-unknown-openbsd)
running: "/data/semarie/build-rust/install_dir/beta/bin/cargo" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
"/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/unstable-book-gen/Cargo.toml" "--message-format" "json-render-diagnostics"
error: failed to run `rustc` to learn about target-specific information
Caused by:
process didn't exit successfully: `/data/semarie/build-rust/build_dir/build/bootstrap/debug/rustc - --crate-name ___ --print=file-names --cfg=bootstrap -Csymbol-mangling-version=v0 -Zmacro-backtrace -Clink-args=-Wl,-z,origin
'-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Csplit-debuginfo=off -Ztls-model=initial-exec --target x86_64-unknown-openbsd --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro
--print=sysroot --print=cfg` (exit status: 1)
--- stdout
Did not run successfully: exit status: 1
"/data/semarie/build-rust/install_dir/beta/bin/rustc" "-" "--crate-name" "___" "--print=file-names" "--cfg=bootstrap" "-Csymbol-mangling-version=v0" "-Zmacro-backtrace" "-Clink-args=-Wl,-z,origin" "-Clink-args=-Wl,-rpath,$ORIGIN/../lib"
"-Csplit-debuginfo=off" "-Ztls-model=initial-exec" "--target" "x86_64-unknown-openbsd" "--crate-type" "bin" "--crate-type" "rlib" "--crate-type" "dylib" "--crate-type" "cdylib" "--crate-type" "staticlib" "--crate-type" "proc-macro"
"--print=sysroot" "--print=cfg" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Wsemicolon_in_expressions_from_macros" "-Dwarnings" "--sysroot" "/data/semarie/build-rust/install_dir/beta"
-------------
--- stderr
error: `-Csplit-debuginfo` is unstable on this platform
command did not execute successfully: "/data/semarie/build-rust/install_dir/beta/bin/cargo.bin" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
```
I am suspecting that all unix might be affected, but I am unsure about the right conditional to use. so I only added "openbsd" target inside it as I am able to test it.
rustc nightly built correctly with this PR on openbsd.
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
Remove DebbugerScripts from bootstrap CLI
This PR implements #95992 and removes the debugger scripts from the bootstrap CLI.
I could not find a lot of documentation on the bootstrap binary so perhaps there's still some documentation to be updated.
after #95612, only linux and windows target are build with -Zunstable-options,
but others platforms might use -Csplit-debuginfo
add "openbsd" target in the list of platforms using it.
Make it possible to write doctests for bootstrap
This probably isn't super useful in practice, but it was easy to fix
and avoids confusing errors about mismatched versions between beta and the default toolchain.
This probably isn't super useful in practice, but it was easy to fix
and avoids confusing errors about mismatched versions between beta and the default toolchain.
Move `download-ci-llvm` out of bootstrap.py
This is ready for review. It has been tested on Windows, Linux, and NixOS.
The second commit ports the changes from https://github.com/rust-lang/rust/pull/95234 to Rust; I can remove it if desired.
Helps with https://github.com/rust-lang/rust/issues/94829.
As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that.
`@rustbot` label +A-rustbuild
Add type_name info to [TIMING] log output
Adds type_name to the [TIMING] log output:
```
[TIMING] (bootstrap::compile::Sysroot) Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } } -- 0.020
[TIMING] (bootstrap::builder::Builder::sysroot_libdir::Libdir) Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } -- 0.007
```
Not sure if that's the best way to format it. Thought about replacing the struct's name with the type_name output, but that feels kind of hacky?
Closes#96060
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
Improve bootstrap tests
- Don't checkout submodules in bootstrap tests
This doesn't cause any tests to fail, and can greatly speed them up.
- Add a test for --exclude test::XXX
I didn't know that the `test::` syntax was valid before, and it doesn't seem to be documented anywhere. Add a test so it doesn't regress accidentally, and as executable documentation.
This also moves the `exclude` tests out of `dist`, to avoid assertion errors when the `cmd` passed to configure didn't match the `subcommand` used.
- Use run_build helper consistently across most bootstrap tests
This is not super important to do, but the consistency is nice.
I didn't change any tests that call `configure("dist")` and then override the subcommand - doing
that at all is pretty sketchy, but I don't want to mess with it while already doing a refactor.
Found while working on the "one call to Step for all paths" change mentioned in https://github.com/rust-lang/rust/pull/95503#issuecomment-1105914384, but independent of that work.
cc `@pietroalbini` for the `--exclude` test, git blame says you added support for it originally.
Add `x {check,build,doc} {compiler,library}` aliases.
While working on https://github.com/rust-lang/rust/pull/95503, I realized that it will interfere with existing command lines:
Currently people run `x build library/std` expecting it to "add all library crates to the sysroot",
but after that change, it will *only* build `libstd` and its dependencies (and add them to the sysroot), not libtest or libproc_macro.
That will work for local testing in most cases, but could be confusing. Even if not, though, I think `x build library` is more clear about what actually happens than the current `x build library/std`.
The intended end goal is something like:
- For check/build/doc, we have library + compiler aliases, which correspond to basically "most possible" for that piece. This is the intended path of entry (rather than library/test or similar as today) for when you just want the thing to work -- for example, getting a compiler that is "crates.io-compatible" would be roughly `x.py build library`). #95504
- Specific crate invocations build up to that crate, which means that if you don't care about tests you probably want x.py build library/proc_macro or library/std for faster build times. #95503
Note that this is already implemented today for the `doc` command and seems to work pretty well in practice.
I plan to change the dev-guide and various instructions in the README to `build library` once this is merged.
`@rustbot` label +A-rustbuild
