Don't download cargo twice when download-rustc is set
Previously, this caused a bug on NixOS:
1. bootstrap.py would download and patch stage0/cargo
2. bootstrap.py would download nightly cargo, but extract it to
stage0/cargo instead of ci-rustc/cargo. It would still try (and fail) to patch ci-rustc/cargo.
3. bootstrap.py would fail to build rustbuild because stage0/cargo
wasn't patched.
The "proper" fix is to extract nightly cargo to ci-rustc instead, but it
doesn't seem to be necessary at all, so this just skips downloading it
instead.
Fixes https://github.com/rust-lang/rust/issues/84702
Allow running `x.py test --stage 2 src/tools/linkchecker` with `download-rustc = true`
Previously, the LD_LIBRARY_PATH for the linkchecker looked like
`build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib`, because the linkchecker depends on the master copy of the standard library. This is true, but doesn't include the library path for the compiler libraries:
```
/home/joshua/src/rust/rust/build/x86_64-unknown-linux-gnu/stage1-tools-bin/error_index_generator: error while loading shared libraries: libLLVM-12-rust-1.53.0-nightly.so: cannot open shared object file: No such file or directory
```
That file is in
`build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-12-rust-1.53.0-nightly.so`,
which wasn't included in the dynamic path. This adds `build/x86_64-unknown-linux-gnu/stage1/lib` to the dynamic path for the linkchecker.
Previously, this caused a bug on NixOS:
1. bootstrap.py would download and patch stage0/cargo
2. bootstrap.py would download nightly cargo, but extract it to
stage0/cargo instead of ci-rustc/cargo.
3. bootstrap.py would fail to build rustbuild because stage0/cargo
wasn't patched.
The "proper" fix is to extract nightly cargo to ci-rustc instead, but it
doesn't seem to be necessary at all, so this just skips downloading it
instead.
Implement `x.py test src/tools/clippy --bless`
- Add clippy_dev to the rust workspace
Before, it would give an error that it wasn't either included or
excluded from the workspace:
```
error: current package believes it's in a workspace when it's not:
current: /home/joshua/rustc/src/tools/clippy/clippy_dev/Cargo.toml
workspace: /home/joshua/rustc/Cargo.toml
this may be fixable by adding `src/tools/clippy/clippy_dev` to the `workspace.members` array of the manifest located at: /home/joshua/rustc/Cargo.toml
Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
```
- Change clippy's copy of compiletest not to special-case
rust-lang/rust. Using OUT_DIR confused `clippy_dev` and it couldn't find
the test outputs. This is one of the reasons why `cargo dev bless` used
to silently do nothing (the others were that `CARGO_TARGET_DIR` and
`PROFILE` weren't set appropriately).
- Run clippy_dev on test failure
I tested this by removing a couple lines from a stderr file, and they
were correctly replaced.
- Fix clippy_dev warnings
Set `backtrace-on-ice` by default for compiler and codegen profiles
If there's an ICE while bootstrapping, it's most likely because of a change to the compiler.
Don't rebuild rustdoc and clippy after checking bootstrap
This works by unconditionally passing -Z unstable-options to the
compiler. This has no affect in practice since bootstrap doesn't use
`deny(rustc::internal)`.
Fixes https://github.com/rust-lang/rust/issues/82461.
r? ```@Mark-Simulacrum```
- Add clippy_dev to the rust workspace
Before, it would give an error that it wasn't either included or
excluded from the workspace:
```
error: current package believes it's in a workspace when it's not:
current: /home/joshua/rustc/src/tools/clippy/clippy_dev/Cargo.toml
workspace: /home/joshua/rustc/Cargo.toml
this may be fixable by adding `src/tools/clippy/clippy_dev` to the `workspace.members` array of the manifest located at: /home/joshua/rustc/Cargo.toml
Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
```
- Change clippy's copy of compiletest not to special-case
rust-lang/rust. Using OUT_DIR confused `clippy_dev` and it couldn't find
the test outputs. This is one of the reasons why `cargo dev bless` used
to silently do nothing (the others were that `CARGO_TARGET_DIR` and
`PROFILE` weren't set appropriately).
- Run clippy_dev on test failure
I tested this by removing a couple lines from a stderr file, and they
were correctly replaced.
- Fix clippy_dev warnings
bootstrap: use bash on illumos to run install scripts
The default illumos shell ("sh" in the default PATH) is ksh93, rather
than bash, and does not support constructs like "local" that came from
bash. The bootstrap function for invoking "install.sh" scripts should
use "bash" explicitly there to avoid issues.
This works by unconditionally passing -Z unstable-options to the
compiler. This has no affect in practice since bootstrap doesn't use
`deny(rustc::internal)`.
Previously, the LD_LIBRARY_PATH for the linkchecker looked like
`build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib`, because the linkchecker depends on the master copy of the standard library. This is true, but doesn't include the library path for the compiler libraries:
```
/home/joshua/src/rust/rust/build/x86_64-unknown-linux-gnu/stage1-tools-bin/error_index_generator: error while loading shared libraries: libLLVM-12-rust-1.53.0-nightly.so: cannot open shared object file: No such file or directory
```
That file is in
`build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-12-rust-1.53.0-nightly.so`,
which wasn't included in the dynamic path. This adds `build/x86_64-unknown-linux-gnu/stage1/lib` to the dynamic path for the linkchecker.
bootstrap: Restore missing --bulk-dirs for rust-docs, rustc-docs
The `--bulk-dirs` argument was removed for rust-docs in commit c768ce1384 and rustc-docs in commit 8ca46fc7a8 (#79788), presumably by mistake; that slowed down installation of rust-docs from under a second to some twenty *minutes*. Restoring `--bulk-dirs` reverses this slowdown.
Fixes#80684.
Cc `@pietroalbini.`
Support `x.py doc std --open`
I usually run this command:
```
./x.py doc std --stage 1 --jobs 8
```
Then I gave a try to `--open` and realized it wasn't working. I finally realized it was simply because it was only handling paths starting with `library`. This PR allows to handle both kinds of paths.
cc ``@jyn514``
r? ``@Mark-Simulacrum``
The --bulk-dirs argument was removed for rust-docs in commit
c768ce1384 and rustc-docs in commit
8ca46fc7a8 (#79788), presumably by
mistake; that slowed down installation of rust-docs from under a
second to some twenty *minutes*. Restoring --bulk-dirs reverses this
slowdown.
Fixes#80684.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
The default illumos shell ("sh" in the default PATH) is ksh93, rather
than bash, and does not support constructs like "local" that came from
bash. The bootstrap function for invoking "install.sh" scripts should
use "bash" explicitly there to avoid issues.
Fix NixOS patching
Moving the `.nix-deps` has resulted in rpath links being broken and
therefore bootstrap on NixOS broken entirely.
This PR still produces a `.nix-deps` but only for the purposes of
producing a gc root. We rpath a symlink-resolved result instead.
For purposes of simplicity we also use joinSymlink to produce a single
merged output directory so that we don't need to update multiple
locations every time we add a library or something.
Fixes a regression from https://github.com/rust-lang/rust/pull/82739.
Moving the `.nix-deps` has resulted in rpath links being broken and
therefore bootstrap on NixOS broken entirely.
This PR still produces a `.nix-deps` but only for the purposes of
producing a gc root. We rpath a symlink-resolved result instead.
For purposes of simplicity we also use joinSymlink to produce a single
merged output directory so that we don't need to update multiple
locations every time we add a library or something.
reduce threads spawned by ui-tests
The test harness already spawns enough tests to keep all cores busy.
Individual tests should keep their own threading to a minimum to avoid context switch overhead.
When running ui tests with lld enabled this shaves about 10% off that testsuite on my machine.
Resolves#81946
the test harness already spawns enough tests for all cores, individual
tests should keep their own threading to a minimum to avoid context switch
overhead
some tests fail with 1 CGU, so explicit compile flags have been added
to keep their old behavior
Cleanup option parsing and config.toml.example
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
7704d35,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/bootstrap.20defaults
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
7704d35acc,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
- Add "as an example" where appropriate
- Link to an issue instead of to ephemeral chats
Rollup of 8 pull requests
Successful merges:
- #83370 (Add `x.py setup tools` which enables `download-rustc` by default)
- #83489 (Properly suggest deref in else block)
- #83734 (Catch a bad placeholder type error for statics in `extern`s)
- #83814 (expand: Do not ICE when a legacy AST-based macro attribute produces and empty expression)
- #83835 (rustdoc: sort search index items for compression)
- #83849 (rustdoc: Cleanup handling of associated items for intra-doc links)
- #83881 (⬆️ rust-analyzer)
- #83885 (Document compiler/ with -Aprivate-intra-doc-links)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Use tikv-jemallocator in rustc/rustdoc in addition to jemalloc-sys when enabled.
In https://github.com/rust-lang/rust/pull/81782 it was mentioned that one reason rustc may benefit from minimalloc is it doesn't use the `sdallocx` api from jemalloc.
Currently, on unix, rust uses jemalloc by importing its symbols to use them with the default, System (libc) global allocator.
This PR switches its global alloc to `tikv-jemallocator`, which correctly uses sized deallocation (https://docs.rs/tikv-jemallocator/0.4.1/src/tikv_jemallocator/lib.rs.html#121-126). `tikv-jemallocator`, as far as I can tell, is a more up-to-date set of bindings to jemalloc than `jemallocator`
The perf results of this pr are in large part due to the version upgrade of jemalloc, but sized deallocation has a non-trivial improvement, particularly to rustdoc.
This pr also includes changes to bootstrap to correctly pass the jemalloc feature through to the rustdoc build
Add `download-rustc = "if-unchanged"`
This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.
This sets `BOOTSTRAP_DOWNLOAD_RUSTC` in bootstrap.py so rustbuild doesn't have to try and replicate its logic.
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum` cc `@camelid`
Use the beta compiler for building bootstrap tools when `download-rustc` is set
## Motivation
This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.
## Implementation
Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.
This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.
Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.
I tested this by rebasing this PR from edeee915b1 over 1c77a1fa3c and confirming that only the bootstrap library itself had to be rebuilt, not any dependencies and not `tidy`. I also tested that a clean build with `x.py build` builds rustdoc exactly once and does no other work, and that `touch src/librustdoc/lib.rs && x.py build` works. `x.py check` still behaves as before (checks using the beta compiler, even if there are changes to `compiler/`).
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum`
## Motivation
This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.
## Implementation
Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.
This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.
Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.
1.52 Cargo adds rust-lang/cargo#8640 which means that cargo will try to purge
the doc directory caches for us. In theory this may mean that we can jettison
the clear_if_dirty for rustdoc versioning entirely, but for now just workaround
the effects of this change in a less principled but more local way.
Adds bootstrap rules to support installing rust-demangler.
When compiling with `-Z instrument-coverage`, the coverage reports are
generated by `llvm-cov`. `llvm-cov` includes a built-in demangler for
C++, and an option to supply an alternate demangler. For Rust, we have
`rust-demangler`, currently used in `rustc` coverage tests.
Fuchsia's toolchain for Rust is built via `./x.py install`. Fuchsia is
adding support for Rust coverage, and we need to include the
`rust-demangler` in the installed `bin` directory.
Configured rust-demangler as an in-tree extended tool.
Added tests to support `./x.py test rust-demangler`.
Install with extended tools by default only if `profiler = true`.
We don't need to complain to the user about linkcheck having different
hosts and targets when it is already excluded. This can be achieved by
moving the check to when the step is run instead of in should_run.
Riscv64linux Test fixes
Get tests passing again using the riscv64gc-unknown-linux-gnu docker image.
Test with
```
src/ci/docker/run.sh riscv64gc-linux
```
## linkcheck
Linkcheck tests that interdocument links in the documentation are correct. Some interdocument links go between rustc and tools (such as rustdoc and cargo). When cross compiling, rustc is built for the host while some tools are built for the target. This goes for the documentation too. Because of this, links in the rustc documentation reffering to cargo or rustdoc documentation look broken.
This issue is worked around by disabling linkcheck for cross compilation builds.
## run-make tests
#78911 seems to happen because `--target` was not passed to `rustc`, but the target linker was specified, causing the target linker to be called with options intended for the host.
Resolves#78911
In a separate issue, `issue-36710` was trying to run a binary built for the target on the host system. This will not work for any platform using `remote-test-server`/`client` (such as riscv64). I don't know of a way of skipping those platforms specifically, so I set this test to skip only on riscv64 for now.
When we cross compile, some things (and their documentation) are built
for the host (e.g. rustc), while others (and their documentation) are built
for the target. This generated documentation will have broken links
between documentation for different platforms e.g. between rustc and
cargo.
