Speed up bootstrap a little.
The bootstrap script was calling `cargo metadata` 3 times (or 6 with `-v`). This is a very expensive operation, and this attempts to avoid the extra calls. On my system, a simple command like `./x.py test -h -v` goes from about 3 seconds to 0.4.
An overview of the changes:
- Call `cargo metadata` only once with `--no-deps`. Optional dependencies are filtered in `in_tree_crates` (handling `profiler_builtins` and `rustc_codegen_llvm` which are driven by the config).
- Remove a duplicate call to `metadata::build` when using `-v`. I'm not sure why it was there, it looks like a mistake or vestigial from previous behavior.
- Remove check for `_shim`, I believe all the `_shim` crates are now gone.
- Remove check for `rustc_` and `*san` for `test::Crate::should_run`, these are no longer dependencies in the `test` tree.
- Use relative paths in `./x.py test -h -v` output.
- Some code cleanup (remove unnecessary `find_compiler_crates`, etc.).
- Show suite paths (`src/test/ui/...`) in `./x.py test -h -v` output.
- Some doc comments.
bootstrap/install.rs: support a nonexistent `prefix` in `x.py install`
PR #49778 introduced fs::canonicalize() which fails for a nonexistent path.
This is a surprise for someone used to GNU Autotools' configure which can create any necessary intermediate directories in prefix.
This change makes it run fs::create_dir_all() before canonicalize().
bootstrap: read config from $RUST_BOOTSTRAP_CONFIG
This PR modifies bootstrap so that `config.toml` is read first from `RUST_BOOTSTRAP_CONFIG`, then `--config` and finally `config.toml` in the current directory.
This is a subjective change, intended to improve the ergnomics when using "development shells" for rustc development (for example, using tools such as Nix) which set environment variables to ensure a reproducible environment (these development shells can then be version controlled, e.g. [my rustc shell](6b74a5c170/nix/shells/rustc.nix)). By optionally reading `config.toml` from an environment variable, a `config.toml` can be defined in the development shell and a path to it exposed in the `RUST_BOOTSTRAP_CONFIG` environment variable - avoiding the need to manually symlink the contents of this file to `config.toml` in the working directory.
Create self-contained directory and move there some of external binaries/libs
One of the steps to reach design described in https://github.com/rust-lang/rust/issues/68887#issuecomment-633048380
This PR moves things around and allows link code to handle the new directory structure.
[RISC-V] Do not force frame pointers
We have been seeing some very inefficient code that went away when using
`-Cforce-frame-pointers=no`. For instance `core::ptr::drop_in_place` at
`-Oz` was compiled into a function which consisted entirely of saving
registers to the stack, then using the frame pointer to restore the same
registers (without any instructions between the prolog and epilog).
The RISC-V LLVM backend supports frame pointer elimination, so it makes
sense to allow this to happen when using Rust. It's not clear to me that
frame pointers have ever been required in the general case.
In rust-lang/rust#61675 it was pointed out that this made reassembling
stack traces easier, which is true, but there is a code generation
option for forcing frame pointers, and I feel the default should not be
to require frame pointers, given it demonstrably makes code size worse
(around 10% in some embedded applications).
The kinds of targets mentioned in rust-lang/rust#61675 are popular, but
should not dictate that code generation should be worse for all RISC-V
targets, especially as there is a way to use CFI information to
reconstruct the stack when the frame pointer is eliminated. It is also
a misconception that `fp` is always used for the frame pointer. `fp` is
an ABI name for `x8` (aka `s0`), and if no frame pointer is required,
`x8` may be used for other callee-saved values.
---
I am partly posting this to get feedback from @fintelia who introduced the change to require frame pointers, and @hanna-kruppe who had issues with the original PR. I would understand if we wanted to remove this setting on only a subset of RISC-V targets, but my preference would be to remove this setting everywhere.
There are more details on the code size savings seen in Tock here: https://github.com/tock/tock/pull/1660
Rollup of 8 pull requests
Successful merges:
- #73237 (Check for overflow in DroplessArena and align returned memory)
- #73339 (Don't run generator transform when there's a TyErr)
- #73372 (Re-order correctly the sections in the sidebar)
- #73373 (Use track caller for bug! macro)
- #73380 (Add more info to `x.py build --help` on default value for `-j JOBS`.)
- #73381 (Fix typo in docs of std::mem)
- #73389 (Use `Ipv4Addr::from<[u8; 4]>` when possible)
- #73400 (Fix forge-platform-support URL)
Failed merges:
r? @ghost
Avoid prematurely recording toolstates
When we're running with dry_run enabled (i.e. all builds do this initially), we're
guaranteed to save of a toolstate of TestFail for tools that aren't tested. In practice,
we do test tools as well, so for those tools we would initially record them as being
TestPass, and then later on re-record the correct state after actually testing them.
However, this would not work well if the build failed for whatever reason (e.g. panicking
in bootstrap, or as was the case in #73097, clippy failing to test successfully), we would
just go on believing that things passed when they in practice did not.
This commit also adjusts saving toolstate to never record clippy explicitly (otherwise, it
would be recorded when building it); eventually that'll likely move to other tools as well
but not yet. This is deemed simpler than checking everywhere we generically save
toolstate.
We also move clippy out of the "toolstate" no-fail-fast build into a separate x.py
invocation; this should no longer be technically required but provides the nice state of
letting us check toolstate for all tools and only then check clippy (giving full results
on every build).
r? @oli-obk
Supercedes #73275, also fixes#73274
This commit modifies bootstrap so that `config.toml` is read first from
`RUST_BOOTSTRAP_CONFIG`, then `--config` and finally `config.toml` in the
current directory.
This is a subjective change, intended to improve the ergnomics when
using "development shells" for rustc development (for example, using tools
such as Nix) which set environment variables to ensure a reproducible
environment (these development shells can then be version controlled). By
optionally reading `config.toml` from an environment variable, a `config.toml`
can be defined in the development shell and a path to it exposed in the
`RUST_BOOTSTRAP_CONFIG` environment variable - avoiding the need to manually
symlink the contents of this file to `config.toml` in the working
directory.
Signed-off-by: David Wood <david@davidtw.co>
When we're running with dry_run enabled (i.e. all builds do this initially), we're
guaranteed to save of a toolstate of TestFail for tools that aren't tested. In practice,
we do test tools as well, so for those tools we would initially record them as being
TestPass, and then later on re-record the correct state after actually testing them.
However, this would not work well if the build failed for whatever reason (e.g. panicking
in bootstrap, or as was the case in 73097, clippy failing to test successfully), we would
just go on believing that things passed when they in practice did not.
This commit also adjusts saving toolstate to never record clippy explicitly (otherwise, it
would be recorded when building it); eventually that'll likely move to other tools as well
but not yet. This is deemed simpler than checking everywhere we generically save
toolstate.
We also move clippy out of the "toolstate" no-fail-fast build into a separate x.py
invocation; this should no longer be technically required but provides the nice state of
letting us check toolstate for all tools and only then check clippy (giving full results
on every build).
PR #49778 introduced fs::canonicalize() which fails for a nonexistent path.
This is a surprise for someone used to GNU Autotools' configure which can create any necessary intermediate directories in prefix.
This change makes it run fs::create_dir_all() before canonicalize().
Implement new gdb/lldb pretty-printers
Reopened#60826
This PR replaces current gdb and lldb pretty-printers with new ones that were originally written for [IntelliJ Rust](https://github.com/intellij-rust/intellij-rust/tree/master/prettyPrinters).
The current state of lldb pretty-printers is poor, because [they don't use synthetic children](https://github.com/rust-lang/rust/issues/55586#issuecomment-436610063). When I started to reimplement lldb pretty-printers with synthetic children support, I've found current version strange and hard to support. I think `debugger_pretty_printers_common.py` is overkill, so I got rid of it.
The new pretty-printers have to support all types supported by current pretty-printers, and also support `Rc`, `Arc`, `Cell`, `Ref`, `RefCell`, `RefMut`, `HashMap`, `HashSet`.
Fixes#56252
Remove vestigial CI job msvc-aux.
This CI job isn't really doing anything, so it seems prudent to remove it.
For some history:
* This was introduced in #48809 when the msvc job was split in two to keep it under 2 hours (oh the good old days). At the time, this check-aux job did a bunch of things:
* tidy
* src/test/pretty
* src/test/run-pass/pretty
* src/test/run-fail/pretty
* src/test/run-pass-valgrind/pretty
* src/test/run-pass-fulldeps/pretty
* src/test/run-fail-fulldeps/pretty
* Tidy was removed in #60777.
* run-pass and run-pass-fulldeps moved to UI in #63029
* src/test/pretty removed in #58140
* src/test/run-fail moved to UI in #71185
* run-fail-fulldeps removed in #51285
Over time through attrition, the job was left with one lonely thing: `src/test/run-pass-valgrind/pretty`. And of course, this wasn't actually running the "pretty" tests. The normal `run-pass-valgrind` tests ran, and then when it tried to run in "pretty" mode, all the tests were ignored because compiletest thought nothing had changed (apparently compiletest isn't fingerprinting the mode? Needs more investigation…). `run-pass-valgrind` is already being run as part of `x86_64-msvc-1`, so there's no need to run it here.
I've taken the liberty of removing `src/test/run-pass-valgrind/pretty` as a distinct test. I'm guessing from the other PR's that the pretty tests should now live in `src/test/pretty`, and that the team has moved away from doing pretty tests on other parts of the `src/test` tree.
This should run much faster.
There are also some drive-by cleanups here to try to simplify things.
Also, the paths for in-tree crates are now displayed as relative
in `x.py test -h -v`.
x.py: do not build Miri by default on stable/beta
Fixes https://github.com/rust-lang/rust/issues/73117
Do I need to do anything to make sure Miri is still built by the tools CI builder? Are there other tools that should be off-by-default?
Also, unfortunately the `DEFAULT` associated const has no doc comment, so I have no idea what it does, or why there are semmingly two places where the default build of tools is controlled.
Use the built cargo for cargotest.
cargotest was using the beta (bootstrap) cargo. This changes it so that it will use the locally built cargo. This is intended to provide a sort of smoke test to ensure Cargo is functional. This *shouldn't* have any real impact on the CI build time. The cargotest job also happens to run cargo's testsuite, so it should already be building cargo.
Note: This will fail until #73266 is merged.
Enable AVR as a Tier 3 target upstream
Tracking issue: #44052.
Things intentionally left out of the initial upstream:
* The `target_cpu` flag
I have made the cleanup suggestions by @jplatte and @jplatte in 043550d9db.
Anybody feel free to give the branch a test and see how it fares, or make suggestions on the code patch itself.
Try_run must only be used if toolstate is populated
Clippy's tests were failing the build, but that failure was ignored in favor of checking toolstate. This is the correct behavior for toolstate-checked tools, but Clippy no longer updates its toolstate status as it should always build.
The previous PR of this kind didn't catch this as I expected x.py failures to always lead to a non-successful build in CI, but that's not the case specifically for tool testing.
Replace old GDB and LLDB pretty-printers with new ones
which were originally written for IntelliJ Rust.
New LLDB pretty-printers support synthetic children.
New GDB/LLDB pretty-printers support all Rust types
supported by old pretty-printers, and also support:
Rc, Arc, Cell, Ref, RefCell, RefMut, HashMap, HashSet.
Clippy's tests were failing the build, but that failure was ignored in favor of
checking toolstate. This is the correct behavior for toolstate-checked tools,
but Clippy no longer updates its toolstate status as it should always build.
Count the beta prerelease number just from master
We were computing a merge-base between the remote beta and master
branches, but this was giving incorrect answers for the first beta if
the remote hadn't been pushed yet. For instance, `1.45.0-beta.3359`
corresponds to the number of merges since the 1.44 beta, but we really
want just `.1` for the sole 1.45 beta promotion merge.
We don't really need to query the remote beta at all -- `master..HEAD`
suffices if we assume that we're on the intended beta branch already.
We were computing a merge-base between the remote beta and master
branches, but this was giving incorrect answers for the first beta if
the remote hadn't been pushed yet. For instance, `1.45.0-beta.3359`
corresponds to the number of merges since the 1.44 beta, but we really
want just `.1` for the sole 1.45 beta promotion merge.
We don't really need to query the remote beta at all -- `master..HEAD`
suffices if we assume that we're on the intended beta branch already.
Make remote-test-client and remote-test-server compatible with windows
`compiletest` and `remote-test-client`:
The command line for `remote-test-client` was changed slightly to allow cross-platform compatible paths. The old way of supplying the support libs was by joining their paths with the executable path
with `:`. This caused Windows-style paths to be split after the directory letter. Now, the number of support libs is provided as a parameter as well, and the support lib paths are split off from the regular args in the client.
`remote-test-server`:
- Marked Unix-only parts as such and implemented Windows alternatives
- On Windows `LD_LIBRARY_PATH` doesn't exist. Libraries are loaded from `PATH` though, so that's the way around it.
- Tiny cleanup: `Command::args`/`envs` instead of manually looping over them
- The temp path for Windows has to be set via environment variable, since there isn't a global temp directory that would work on every machine (as a static string)
We have been seeing some very inefficient code that went away when using
`-Cforce-frame-pointers=no`. For instance `core::ptr::drop_in_place` at
`-Oz` was compiled into a function which consisted entirely of saving
registers to the stack, then using the frame pointer to restore the same
registers (without any instructions between the prolog and epilog).
The RISC-V LLVM backend supports frame pointer elimination, so it makes
sense to allow this to happen when using Rust. It's not clear to me that
frame pointers have ever been required in the general case.
In rust-lang/rust#61675 it was pointed out that this made reassembling
stack traces easier, which is true, but there is a code generation
option for forcing frame pointers, and I feel the default should not be
to require frame pointers, given it demonstrably makes code size worse
(around 10% in some embedded applications).
The kinds of targets mentioned in rust-lang/rust#61675 are popular, but
should not dictate that code generation should be worse for all RISC-V
targets, especially as there is a way to use CFI information to
reconstruct the stack when the frame pointer is eliminated. It is also
a misconception that `fp` is always used for the frame pointer. `fp` is
an ABI name for `x8` (aka `s0`), and if no frame pointer is required,
`x8` may be used for other callee-saved values.
This commit does ensure that the standard library is built with unwind
tables, so that users do not need to rebuild the standard library in
order to get a backtrace that includes standard library calls (which is
the original reason for forcing frame pointers).
Update Clippy, RLS, and rustfmt
r? @Dylan-DPC
This makes Clippy test-pass again: 3089c3b
Otherwise this includes bugfixes and a few new lints.
Fixes#72231Fixes#72232
Since rustc-ap-* v659 we now need to set CFG_RELEASE for
rustc-ap-rustc_attr for `#[cfg(version(...))]` to work.
Co-authored-by: Eric Huss <ehuss@users.noreply.github.com>
