MinGW driver for COFF LLD doesn't currently translate GNU-style `--threads=N` to native `/threads:N`, so we have to pass the option in its native form to avoid an error.
Also pass the `threads` flag to lld-link as well
Don't download cargo twice when download-rustc is set
Previously, this caused a bug on NixOS:
1. bootstrap.py would download and patch stage0/cargo
2. bootstrap.py would download nightly cargo, but extract it to
stage0/cargo instead of ci-rustc/cargo. It would still try (and fail) to patch ci-rustc/cargo.
3. bootstrap.py would fail to build rustbuild because stage0/cargo
wasn't patched.
The "proper" fix is to extract nightly cargo to ci-rustc instead, but it
doesn't seem to be necessary at all, so this just skips downloading it
instead.
Fixes https://github.com/rust-lang/rust/issues/84702
Allow running `x.py test --stage 2 src/tools/linkchecker` with `download-rustc = true`
Previously, the LD_LIBRARY_PATH for the linkchecker looked like
`build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib`, because the linkchecker depends on the master copy of the standard library. This is true, but doesn't include the library path for the compiler libraries:
```
/home/joshua/src/rust/rust/build/x86_64-unknown-linux-gnu/stage1-tools-bin/error_index_generator: error while loading shared libraries: libLLVM-12-rust-1.53.0-nightly.so: cannot open shared object file: No such file or directory
```
That file is in
`build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-12-rust-1.53.0-nightly.so`,
which wasn't included in the dynamic path. This adds `build/x86_64-unknown-linux-gnu/stage1/lib` to the dynamic path for the linkchecker.
Previously, this caused a bug on NixOS:
1. bootstrap.py would download and patch stage0/cargo
2. bootstrap.py would download nightly cargo, but extract it to
stage0/cargo instead of ci-rustc/cargo.
3. bootstrap.py would fail to build rustbuild because stage0/cargo
wasn't patched.
The "proper" fix is to extract nightly cargo to ci-rustc instead, but it
doesn't seem to be necessary at all, so this just skips downloading it
instead.
Implement `x.py test src/tools/clippy --bless`
- Add clippy_dev to the rust workspace
Before, it would give an error that it wasn't either included or
excluded from the workspace:
```
error: current package believes it's in a workspace when it's not:
current: /home/joshua/rustc/src/tools/clippy/clippy_dev/Cargo.toml
workspace: /home/joshua/rustc/Cargo.toml
this may be fixable by adding `src/tools/clippy/clippy_dev` to the `workspace.members` array of the manifest located at: /home/joshua/rustc/Cargo.toml
Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
```
- Change clippy's copy of compiletest not to special-case
rust-lang/rust. Using OUT_DIR confused `clippy_dev` and it couldn't find
the test outputs. This is one of the reasons why `cargo dev bless` used
to silently do nothing (the others were that `CARGO_TARGET_DIR` and
`PROFILE` weren't set appropriately).
- Run clippy_dev on test failure
I tested this by removing a couple lines from a stderr file, and they
were correctly replaced.
- Fix clippy_dev warnings
Set `backtrace-on-ice` by default for compiler and codegen profiles
If there's an ICE while bootstrapping, it's most likely because of a change to the compiler.
Don't rebuild rustdoc and clippy after checking bootstrap
This works by unconditionally passing -Z unstable-options to the
compiler. This has no affect in practice since bootstrap doesn't use
`deny(rustc::internal)`.
Fixes https://github.com/rust-lang/rust/issues/82461.
r? ```@Mark-Simulacrum```
- Add clippy_dev to the rust workspace
Before, it would give an error that it wasn't either included or
excluded from the workspace:
```
error: current package believes it's in a workspace when it's not:
current: /home/joshua/rustc/src/tools/clippy/clippy_dev/Cargo.toml
workspace: /home/joshua/rustc/Cargo.toml
this may be fixable by adding `src/tools/clippy/clippy_dev` to the `workspace.members` array of the manifest located at: /home/joshua/rustc/Cargo.toml
Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
```
- Change clippy's copy of compiletest not to special-case
rust-lang/rust. Using OUT_DIR confused `clippy_dev` and it couldn't find
the test outputs. This is one of the reasons why `cargo dev bless` used
to silently do nothing (the others were that `CARGO_TARGET_DIR` and
`PROFILE` weren't set appropriately).
- Run clippy_dev on test failure
I tested this by removing a couple lines from a stderr file, and they
were correctly replaced.
- Fix clippy_dev warnings
bootstrap: use bash on illumos to run install scripts
The default illumos shell ("sh" in the default PATH) is ksh93, rather
than bash, and does not support constructs like "local" that came from
bash. The bootstrap function for invoking "install.sh" scripts should
use "bash" explicitly there to avoid issues.
This works by unconditionally passing -Z unstable-options to the
compiler. This has no affect in practice since bootstrap doesn't use
`deny(rustc::internal)`.
Previously, the LD_LIBRARY_PATH for the linkchecker looked like
`build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib`, because the linkchecker depends on the master copy of the standard library. This is true, but doesn't include the library path for the compiler libraries:
```
/home/joshua/src/rust/rust/build/x86_64-unknown-linux-gnu/stage1-tools-bin/error_index_generator: error while loading shared libraries: libLLVM-12-rust-1.53.0-nightly.so: cannot open shared object file: No such file or directory
```
That file is in
`build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-12-rust-1.53.0-nightly.so`,
which wasn't included in the dynamic path. This adds `build/x86_64-unknown-linux-gnu/stage1/lib` to the dynamic path for the linkchecker.
bootstrap: Restore missing --bulk-dirs for rust-docs, rustc-docs
The `--bulk-dirs` argument was removed for rust-docs in commit c768ce1384 and rustc-docs in commit 8ca46fc7a8 (#79788), presumably by mistake; that slowed down installation of rust-docs from under a second to some twenty *minutes*. Restoring `--bulk-dirs` reverses this slowdown.
Fixes#80684.
Cc `@pietroalbini.`
Support `x.py doc std --open`
I usually run this command:
```
./x.py doc std --stage 1 --jobs 8
```
Then I gave a try to `--open` and realized it wasn't working. I finally realized it was simply because it was only handling paths starting with `library`. This PR allows to handle both kinds of paths.
cc ``@jyn514``
r? ``@Mark-Simulacrum``
The --bulk-dirs argument was removed for rust-docs in commit
c768ce1384 and rustc-docs in commit
8ca46fc7a8 (#79788), presumably by
mistake; that slowed down installation of rust-docs from under a
second to some twenty *minutes*. Restoring --bulk-dirs reverses this
slowdown.
Fixes#80684.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
The default illumos shell ("sh" in the default PATH) is ksh93, rather
than bash, and does not support constructs like "local" that came from
bash. The bootstrap function for invoking "install.sh" scripts should
use "bash" explicitly there to avoid issues.
Fix NixOS patching
Moving the `.nix-deps` has resulted in rpath links being broken and
therefore bootstrap on NixOS broken entirely.
This PR still produces a `.nix-deps` but only for the purposes of
producing a gc root. We rpath a symlink-resolved result instead.
For purposes of simplicity we also use joinSymlink to produce a single
merged output directory so that we don't need to update multiple
locations every time we add a library or something.
Fixes a regression from https://github.com/rust-lang/rust/pull/82739.
Moving the `.nix-deps` has resulted in rpath links being broken and
therefore bootstrap on NixOS broken entirely.
This PR still produces a `.nix-deps` but only for the purposes of
producing a gc root. We rpath a symlink-resolved result instead.
For purposes of simplicity we also use joinSymlink to produce a single
merged output directory so that we don't need to update multiple
locations every time we add a library or something.
reduce threads spawned by ui-tests
The test harness already spawns enough tests to keep all cores busy.
Individual tests should keep their own threading to a minimum to avoid context switch overhead.
When running ui tests with lld enabled this shaves about 10% off that testsuite on my machine.
Resolves#81946
the test harness already spawns enough tests for all cores, individual
tests should keep their own threading to a minimum to avoid context switch
overhead
some tests fail with 1 CGU, so explicit compile flags have been added
to keep their old behavior
Cleanup option parsing and config.toml.example
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
7704d35,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/bootstrap.20defaults
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
7704d35acc,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
- Add "as an example" where appropriate
- Link to an issue instead of to ephemeral chats
Rollup of 8 pull requests
Successful merges:
- #83370 (Add `x.py setup tools` which enables `download-rustc` by default)
- #83489 (Properly suggest deref in else block)
- #83734 (Catch a bad placeholder type error for statics in `extern`s)
- #83814 (expand: Do not ICE when a legacy AST-based macro attribute produces and empty expression)
- #83835 (rustdoc: sort search index items for compression)
- #83849 (rustdoc: Cleanup handling of associated items for intra-doc links)
- #83881 (⬆️ rust-analyzer)
- #83885 (Document compiler/ with -Aprivate-intra-doc-links)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Use tikv-jemallocator in rustc/rustdoc in addition to jemalloc-sys when enabled.
In https://github.com/rust-lang/rust/pull/81782 it was mentioned that one reason rustc may benefit from minimalloc is it doesn't use the `sdallocx` api from jemalloc.
Currently, on unix, rust uses jemalloc by importing its symbols to use them with the default, System (libc) global allocator.
This PR switches its global alloc to `tikv-jemallocator`, which correctly uses sized deallocation (https://docs.rs/tikv-jemallocator/0.4.1/src/tikv_jemallocator/lib.rs.html#121-126). `tikv-jemallocator`, as far as I can tell, is a more up-to-date set of bindings to jemalloc than `jemallocator`
The perf results of this pr are in large part due to the version upgrade of jemalloc, but sized deallocation has a non-trivial improvement, particularly to rustdoc.
This pr also includes changes to bootstrap to correctly pass the jemalloc feature through to the rustdoc build
Add `download-rustc = "if-unchanged"`
This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.
This sets `BOOTSTRAP_DOWNLOAD_RUSTC` in bootstrap.py so rustbuild doesn't have to try and replicate its logic.
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum` cc `@camelid`
Use the beta compiler for building bootstrap tools when `download-rustc` is set
## Motivation
This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.
## Implementation
Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.
This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.
Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.
I tested this by rebasing this PR from edeee915b1 over 1c77a1fa3c and confirming that only the bootstrap library itself had to be rebuilt, not any dependencies and not `tidy`. I also tested that a clean build with `x.py build` builds rustdoc exactly once and does no other work, and that `touch src/librustdoc/lib.rs && x.py build` works. `x.py check` still behaves as before (checks using the beta compiler, even if there are changes to `compiler/`).
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum`
## Motivation
This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.
## Implementation
Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.
This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.
Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.
