Distribute json doc
# Overview
We add a new component, `rust-json-docs`, to distribute the JSON version of rustdoc's output for public compiler crates (i.e. `std`, `alloc`, `proc_macro`, `core` and `test`).
As discussed in #101383, we do not bundle this up as part of the existing `rust-docs` component since `rustdoc`'s JSON format is still unstable.
# Open questions / Doubts
I tried my best, but I never touched this codebase and I couldn't find much documentation on how `dist` works - I pattern-matched existing code, which might have led to some non-sensical choices in the eyes of people more familiar with the codebase. In particular, I am not sure if my choice of adding a new config flag is appropriate or if the decision to build/not build the JSON docs is more appropriately gated by one of the existing flags.
Any suggestion is more than welcome.
Closes#101383
Revert "Copy stage0 binaries into stage0-sysroot"
This reverts PR #101711.
The PR broke the rustc/bootstrap benchmark on rustc-perf, I believe due to the assumption that the stage0 directory exists. Fixing that by just skipping this logic might be reasonable, but I think there's a larger discussion to be had around the right behavior when we don't have a single bin/ directory (when rustc= and cargo= are specified in config.toml). I think it's potentially reasonable to put those binaries (cargo, rustc, rustfmt?) into the bin directory, but for now just want to get us back to a healthy state.
r? `@jyn514` (but would appreciate review from others as this is just a direct revert).
Allow building `rust-analyzer-proc-macro-srv` as a standalone tool
This PR allows building `rust-analyzer-proc-macro-srv` as a standalone tool via `x b proc-macro-srv-cli` (I thought that `x b rust-analyzer-proc-macro-srv` should work, but it doesn't for some reason...). Also this PR adds a copy of `rust-analyzer-proc-macro-srv` binary to `build/{triple}/{stage}/libexec/` when building `rust-analyzer-proc-macro-srv`, so that r-a can pick it up.
This is useful to make r-a (and I assume Intellij IDEA) to expand macros when using a custom, build from source toolchain.
r? ``@jyn514``
[_zulip thread_](https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Frust-analyzer/topic/How.20to.20fix.20.60UnsupportedABI.60.20for.20custom.20toolchains.3F/near/299040175)
Improve handing of env vars during bootstrap process
This CL modifies the handing of env vars during the bootstrap process in two ways:
1. Replaces '-' characters with '_' characters in target names to increase compatibility with different shells
2. Passes Stage0 snapshot compiler related env vars to early invocations of Cargo
Extend list of targets that support dyanmic linking for llvm tools
This commit adds `linux-musl` to the list of targets that support dynamic linking for the LLVM tools.
Pass --cfg=bootstrap for rustdoc for proc_macro crates
This PR does three things:
* First, it passes --cfg=bootstrap on stage 0 for rustdoc invocations on proc_macro crates. This mirrors what we do already for rustc invocations of those, and is needed because cargo doesn't respect RUSTFLAGS or RUSTDOCFLAGS when confronted with a proc macro.
* Second, it marks the bootstrap config variable as expected. This is needed both on later stages where it's not set, but also on stage 0, where it is set.
* Third, it adjusts the comment in the rustc wrapper to better reflect the reason why we set the bootstrap variable as
expected: due to recent changes, setting it as expected
is also required even if the cfg variable is passed: ebf4cc361e .
This was originally added so those doc pages could use the same font
files, but it turns out to be fragile. And those doc pages are just
stubs that link to other pages, so they don't need fancy fonts.
This CL modifies the handing of env vars during the bootstrap process in
two ways:
1. Replaces '-' characters with '_' characters in target names to
increase compatibility with different shells
2. Passes Stage0 snapshot compiler related env vars to early invocations
of Cargo
This commit does three things:
* First, it passes --cfg=bootstrap on stage 0 for rustdoc
invocations on proc_macro crates. This mirrors what we
do already for rustc invocations of those, and is needed
because cargo doesn't respect RUSTFLAGS or RUSTDOCFLAGS
when confronted with a proc macro.
* Second, it marks the bootstrap config variable as expected.
This is needed both on later stages where it's not set,
but also on stage 0, where it is set.
* Third, it adjusts the comment in the rustc wrapper to better
reflect the reason why we set the bootstrap variable as
expected: due to recent changes, setting it as expected
is also required even if the cfg variable is passed: ebf4cc361e .
Replace `check_missing_items.py` with `jsondoclint`
[zulip discussion](https://rust-lang.zulipchat.com/#narrow/stream/266220-rustdoc/topic/check_missing_items.2Epy.20Replacement.2E)
check_missing_items.py was a python script that checked rustdoc json output to make sure all the Id's referenced existed in the JSON index. This PR replaces that with a rust binary (`jsondoclint`) that does the same thing.
### Motivation
1. Easier to change when `rustdoc-json-types` changes, as `jsondoclint` uses the types directly.
2. Better Errors:
- Multiple Errors can be emited for a single crate
- Errors can say where in JSON they occored
```
2:2889:408 not in index or paths, but refered to at '.index."2:2888:104".inner.items[0]'
2:2890:410 not in index or paths, but refered to at '.index."2:2888:104".inner.items[1]'
```
3. Catches more bugs.
- Because matches are exaustive, all posible variants considered for enums
- All Id's checked
- Has already found #101770, #101199 and #100973
- Id type is also checked, so the Id's in a structs fields can only be field items.
4. Allows the possibility of running from `rustdoc::json`, which we should do in a crator run at some point.
cc ``@CraftSpider``
r? ``@GuillaumeGomez``
The `rust-dev` dist component puts binaries in `bootstrap/bin`, but we expected them to be in
`bootstrap/debug` to match cargo's behavior. Rather than making the dist component inconsistent
with other components, make bootstrap slightly smarter and allow using any path as long as all the
binaries are in the same directory.
As a bonus, this greatly simplifies the logic, and makes it possible for the shell scripts to start
avoiding python.
Co-authored-by: Joshua Nelson <github@jyn.dev>
- Default to trying git rev-parse for the root directory
CARGO_MANIFEST_DIR is a path on the build machine, not the running machine.
Don't require this to succeed, to allow building from a tarball; in that case fall back to CARGO_MANIFEST_DIR.
- Set `initial_rustc` to a path based on the path of the running executable, not CARGO_MANIFEST_DIR.
We only reset `initial_rustc` if we're sure this isn't the working tree bootstrap was originally built in,
since I'm paranoid that setting this in other cases will cause things to break;
it's not clear to me when $RUSTC differs from `build/$TARGET/stage0/bin/rustc` (maybe never? but better to be sure).
Instead, only set this when
a) We are not using a custom rustc. If someone has specified a custom rustc we should respect their wishes.
b) We are in a checkout of rust-lang/rust other than the one bootstrap was built in.
- Add a new `bootstrap` component
Originally, we planned to combine this with the `rust-dev` component.
However, I realized that would force LLVM to be redownloaded whenever bootstrap is modified.
LLVM is a much larger download, so split this to get better caching.
- Build bootstrap for all tier 1 and 2 targets
Generate error index with mdbook instead of raw HTML pages
This is a follow-up of https://github.com/rust-lang/rust/pull/100922.
This comes from a remark from ````@estebank```` who said that the search was a nice thing on the previous version and that it wasn't possible anymore. An easy way to come around this limitation was to use `mdbook`, which is what I did here.
Now some explanations on the code. I'll explain how I developed this and why I reached this solution. First I did it very basically by simply setting the source directory and the output directory, generated a `SUMMARY.md` manually which listed all error codes and that was it. Two problems arose from this:
1. A lot of new HTML files were generated at the top level
2. An `index.html` file was generated at the top-level (the summary in short).
So for `1.`, it's not great to have too many files at the top-level as it could create file conflicts more easily. And for `2.`, this is actually a huge issue because <doc.rust-lang.org> generates an `index.html` file with a links to a few different resources, so it should never be overwritten. <s>Unfortunately, `mdbook` **always** generates an `index.html` file so the only solution I could see (except for sending them a contribution, I'll maybe do that later) was to temporaly move a potentially existing `index.html` file and then puts it back once done. For this last part, to ensure that we don't return *before* it has been put back, I wrapped the `mdbook` generation code inside `render_html_inner` which is called from `render_html` which in turn handle the "save" of `index.html`.</s>
EDIT: `mdbook` completely deletes ALL the content in the target directory so I instead generate into a sub directory and then I move the files to the real target directory.
To keep compatibility with the old version, I also put the `<script>` ````@notriddle```` nicely provided in the previous PR only into the `error-index.html` file to prevent unneeded repetition. I didn't use `mdbook` `additional-js` option because the JS is included at the end of all HTML files, which we don't want for two reasons:
1. It's slow.
2. We only want it to be run in `error-index.html` (even if we also ensure in the JS itself too!).
<s>Now the last part: why we generate the summary twice. We actually generate it once to tell `mdbook` what the book will look like and a second time because a create a new chapter content which will actually list all the error codes (with the updated paths).</s>
EDIT: I removed the need for two summaries.
You can test it [here](https://rustdoc.crud.net/imperio/error-index-mdbook/error-index.html).
r? ````@notriddle````
bootstrap: Add llvm-has-rust-patches target option
This is so you can check out an upstream commit in src/llvm-project and
have everything just work.
This simplifies the logic in `is_rust_llvm` a bit; it doesn't need to
check for download-ci-llvm because we would have already errored if both
that and llvm-config were specified on the host platform.
ci: Upgrade non-dist Linux testers from ubuntu:16.04 to 22.04
The main goal of updating to 22.04 is to get away from `llvm.allow-old-toolchain`.
A side benefit is that they can also use the system `cmake` instead of building one.
Fix a bunch of typo
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
Add tier-3 support for powerpc64 and riscv64 openbsd
# powerpc64
- MCP for [powerpc64-unknown-openbsd tier-3 support](https://github.com/rust-lang/compiler-team/issues/551)
- only need to add spec definition in rustc_target
# riscv64
- MCP for [riscv64-unknown-openbsd tier-3 support](https://github.com/rust-lang/compiler-team/issues/552)
- add spec definition in rustc_target
- follow freebsd about avoiding linking with `libatomic`
Fix wrong compiletest filters on Windows
As discussed in [#79334](https://github.com/rust-lang/rust/issues/79334), when calling e.g.
```
python x.py test src/test/ui/expr/compound-assignment/eval-order.rs
```
on Windows, compiletest passes the filter `expr/compound-assignment/eval-order.rs` to libtest, which instead should be `expr\compound-assignment\eval-order.rs`, as that is the file found when collecting tests. This is what I fixed.
I'm not sure how to organize a test for this. Any suggestions?
session: stabilize split debuginfo on linux
Stabilize the `-Csplit-debuginfo` flag...
- ...on Linux for all values of the flag. Split DWARF has been implemented for a few months, hasn't had any bug reports and has had some promising benchmarking for incremental debug build performance.
- ..on other platforms for the default value. It doesn't make any sense that `-Csplit-debuginfo=packed` is unstable on Windows MSVC when that's the default behaviour, but keep the other values unstable.
In Rust for Linux we are using these to make `alloc` a bit
more modular.
A `run-make-fulldeps` test is added for each of them, so that
enabling each of them independently is kept in a compilable state.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
