Update how WASI toolchains are used in CI and bootstrap
This commit updates how the WASI targets are configured with their toolchain. Long ago a `config.toml` option of `wasi-root` was added to enable building with the WASI files produced by wasi-libc. Additionally for CI testing and release building the Rust toolchain has been using a hard-coded commit of wasi-libc which is bundled with the release of the `wasm32-wasip1` target, for example.
Nowadays though the wasi-sdk project, the C/C++ toolchain for WASI, is the go-to solution for compiling/linking WASI code and contains the more-or-less official releases of wasi-libc. This commit migrates CI to using wasi-sdk releases and additionally updates `bootstrap` to recognize when this is configured. This means that with `$WASI_SDK_PATH` configured there's no further configuration necessary to get a working build. Notably this also works better for the new targets of WASI as well, such as `wasm32-wasip2` and `wasm32-wasip1-threads` where the wasi-sdk release now has libraries for all targets bundled within it.
Improve the experience of running Docker locally
When running locally, the absence of the `GITHUB_STEP_SUMMARY` environment variable will lead to the following error:
```
::endgroup::
./src/ci/docker/run.sh: line 349: : No such file or directory
```
I've also changed the output artifacts directory to `obj/$image_name`, allowing me to easily run all images locally. We always encounter various strange issues when modifying the test cases in the `codegen` directory.
r? Kobzol cc `@saethlin`
Previously this command was linting compiler and library together.
As we no longer run clippy on the entire tree unless it's explicitly
requested, we need to update this command by adding `library` path.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
This commit updates how the WASI targets are configured with their
toolchain. Long ago a `config.toml` option of `wasi-root` was added to
enable building with the WASI files produced by wasi-libc. Additionally
for CI testing and release building the Rust toolchain has been using a
hard-coded commit of wasi-libc which is bundled with the release of the
`wasm32-wasip1` target, for example.
Nowadays though the wasi-sdk project, the C/C++ toolchain for WASI, is
the go-to solution for compiling/linking WASI code and contains the
more-or-less official releases of wasi-libc. This commit migrates CI to
using wasi-sdk releases and additionally updates `bootstrap` to
recognize when this is configured. This means that with `$WASI_SDK_PATH`
configured there's no further configuration necessary to get a working
build. Notably this also works better for the new targets of WASI as
well, such as `wasm32-wasip2` and `wasm32-wasip1-threads` where the
wasi-sdk release now has libraries for all targets bundled within it.
ci: test cargo on `aarch64-gnu`
Since `aarch64-unknown-linux-gnu` is a tier-1 target, we should also test cargo on it, especially since cargo's own CI doesn't cover this yet. This might have helped us discover #123733 sooner, which is not a cargo problem but was uncovered by a new cargo test (which we'll have to skip for now). Everything else passes in my local run, so at least we'll have a guard against future regressions.
Enable building tier2 target riscv32im-unknown-none-elf
riscv32im-unknown-none-elf was promoted to tier2 in
https://github.com/rust-lang/rust/pull/117874
but it has not yet been added to the list of build targets.
By adding riscv32im-unknown-none-elf to the list of build targets, this PR enables end-users to install this target via rustup.
[rustdoc] [GUI tests] Make theme switching closer to reality
Better to actually perform actions user do rather than only testing the change through local storage.
As for `browser-ui-test` update: I updated `puppeteer` version (to `0.19.4`) and fixed a bug when displaying the file if it came from an `include`.
r? `@notriddle`
Rollup of 4 pull requests
Successful merges:
- #114788 (impl get_mut_or_init and get_mut_or_try_init for OnceCell and OnceLock)
- #122291 (Stabilize `const_caller_location` and `const_location_fields`)
- #123357 (CI: Redirect stderr to stdout to order GHA logs)
- #123504 (bootstrap: split cargo-miri test into separate Step)
r? `@ghost`
`@rustbot` modify labels: rollup
Support running library tests in Miri
This adds a new bootstrap subcommand `./x.py miri` which can test libraries in Miri. This is in preparation for eventually doing that as part of bors CI, but this PR only adds the infrastructure, and doesn't enable it yet.
`@rust-lang/bootstrap` should this be `x.py test --miri library/core` or `x.py miri library/core`? The flag has the advantage that we don't have to copy all the arguments from `Subcommand::Test`. It has the disadvantage that most test steps just ignore `--miri` and still run tests the regular way. For clippy you went the route of making it a separate subcommand. ~~I went with a flag now as that seemed easier, but I can change this.~~ I made it a new subcommand. Note however that the regular cargo invocation would be `cargo miri test ...`, so `x.py` is still going to be different in that the `test` is omitted. That said, we could also make it `./x.py miri-test` to make that difference smaller -- that's in fact more consistent with the internal name of the command when bootstrap invokes cargo.
`@rust-lang/libs` ~~unfortunately this PR does some unholy things to the `lib.rs` files of our library crates.~~
`@m-ou-se` found a way that entirely avoids library-level hacks, except for some new small `lib.miri.rs` files that hopefully you will never have to touch. There's a new hack in cargo-miri but there it is in good company...
Update to new browser-ui-test version
This new version brings a lot of new internal improvements (mostly around validating the commands input).
It also improved some command names and arguments.
r? `@notriddle`
rustdoc-search: shard the search result descriptions
## Preview
This makes no visual changes to rustdoc search. It's a pure perf improvement.
<details><summary>old</summary>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240317_AiDc61_2EM,240317_AiDcM0_2EN>
Waterfall diagram:
</details>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc2/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240322_BiDcCH_13R,240322_AiDcJY_104>
## Description
r? `@GuillaumeGomez`
The descriptions are, on almost all crates[^1], the majority of the size of the search index, even though they aren't really used for searching. This makes it relatively easy to separate them into their own files.
Additionally, this PR pulls out information about whether there's a description into a bitmap. This allows us to sort, truncate, *then* download.
This PR also bumps us to ES8. Out of the browsers we support, all of them support async functions according to caniuse.
https://caniuse.com/async-functions
[^1]:
<https://microsoft.github.io/windows-docs-rs/>, a crate with
44MiB of pure names and no descriptions for them, is an outlier
and should not be counted. But this PR should improve it, by replacing a long line of empty strings with a compressed bitmap with a single Run section. Just not very much.
## Detailed sizes
```console
$ cat test.sh
set -ex
cp ../search-index*.js search-index.js
awk 'FNR==NR {a++;next} FNR<a-3' search-index.js{,} | awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}' | sed -E "s:\\\\':':g" > search-index.json
jq -c '.t' search-index.json > t.json
jq -c '.n' search-index.json > n.json
jq -c '.q' search-index.json > q.json
jq -c '.D' search-index.json > D.json
jq -c '.e' search-index.json > e.json
jq -c '.i' search-index.json > i.json
jq -c '.f' search-index.json > f.json
jq -c '.c' search-index.json > c.json
jq -c '.p' search-index.json > p.json
jq -c '.a' search-index.json > a.json
du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
$ bash test.sh
+ cp ../search-index1.78.0.js search-index.js
+ awk 'FNR==NR {a++;next} FNR<a-3' search-index.js search-index.js
+ awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}'
+ sed -E 's:\\'\'':'\'':g'
+ jq -c .t search-index.json
+ jq -c .n search-index.json
+ jq -c .q search-index.json
+ jq -c .D search-index.json
+ jq -c .e search-index.json
+ jq -c .i search-index.json
+ jq -c .f search-index.json
+ jq -c .c search-index.json
+ jq -c .p search-index.json
+ jq -c .a search-index.json
+ du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
64K t.json
800K n.json
8.0K q.json
4.0K D.json
16K e.json
192K i.json
544K f.json
4.0K c.json
36K p.json
20K a.json
```
These are, roughly, the size of each section in the standard library (this tool actually excludes libtest, for parsing-json-with-awk reasons, but libtest is tiny so it's probably not important).
t = item type, like "struct", "free fn", or "type alias". Since one byte is used for every item, this implies that there are approximately 64 thousand items in the standard library.
n = name, and that's now the largest section of the search index with the descriptions removed from it
q = parent *module* path, stored parallel to the items within
D = the size of each description shard, stored as vlq hex numbers
e = empty description bit flags, stored as a roaring bitmap
i = parent *type* index as a link into `p`, stored as decimal json numbers; used only for associated types; might want to switch to vlq hex, since that's shorter, but that would be a separate pr
f = function signature, stored as lists of lists that index into `p`
c = deprecation flag, stored as a roaring bitmap
p = parent *type*, stored separately and linked into from `i` and `f`
a = alias, as [[key, value]] pairs
## Search performance
http://notriddle.com/rustdoc-html-demo-11/perf-shard/index.html
For example, in stm32f4:
<table><thead><tr><th>before<th>after</tr></thead>
<tbody><tr><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 617
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 198
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 282
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 426
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 673
```
</td><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 716
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 207
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 289
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 418
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 687
```
</td></tr><tr><td>
```
user: 005.345 s
sys: 002.955 s
wall: 006.899 s
child_RSS_high: 583664 KiB
group_mem_high: 557876 KiB
```
</td><td>
```
user: 004.652 s
sys: 000.565 s
wall: 003.865 s
child_RSS_high: 538696 KiB
group_mem_high: 511724 KiB
```
</td></tr>
</table>
This perf tester is janky and unscientific enough that the apparent differences might just be noise. If it's not an order of magnitude, it's probably not real.
## Future possibilities
* Currently, results are not shown until the descriptions are downloaded. Theoretically, the description-less results could be shown. But actually doing that, and making sure it works properly, would require extra work (we have to be careful to avoid layout jumps).
* More than just descriptions can be sharded this way. But we have to be careful to make sure the size wins are worth the round trips. Ideally, data that’s needed only for display should be sharded while data needed for search isn’t.
* [Full text search](https://internals.rust-lang.org/t/full-text-search-for-rustdoc-and-doc-rs/20427) also needs this kind of infrastructure. A good implementation might store a compressed bloom filter in the search index, then download the full keyword in shards. But, we have to be careful not just of the amount readers have to download, but also of the amount that [publishers](https://gist.github.com/notriddle/c289e77f3ed469d1c0238d1d135d49e1) have to store.
ci: Build gccjit from a git archive
A full `git clone` of GCC includes quite a lot of history, and it's
completely unnecessary for building it in CI. We can use a GitHub
archive URL to get a simple tarball that is much faster to download.
Also, the `gcc-build` directory can be removed after install to reduce
the image size even further.
This is implemented with the freshly-released Wasmtime 19 and should
prevent beta breakage from wasm tests that was observed and fixed
in #122640 again.
A full `git clone` of GCC includes quite a lot of history, and it's
completely unnecessary for building it in CI. We can use a GitHub
archive URL to get a simple tarball that is much faster to download.
Also, the `gcc-build` directory can be removed after install to reduce
the image size even further.
The descriptions are, on almost all crates[^1], the majority
of the size of the search index, even though they aren't really
used for searching. This makes it relatively easy to separate
them into their own files.
This commit also bumps us to ES8. Out of the browsers we support,
all of them support async functions according to caniuse.
https://caniuse.com/async-functions
[^1]:
<https://microsoft.github.io/windows-docs-rs/>, a crate with
44MiB of pure names and no descriptions for them, is an outlier
and should not be counted.
Check library crates for all tier 1 targets in PR CI
Let's try checking all tier 1 targets. Shouldn't take much time.
Not sure if this is the right place to put it or not but let's see if it works first.
Rename `wasm32-wasi-preview1-threads` to `wasm32-wasip1-threads`
This commit renames the current `wasm32-wasi-preview1-threads` target to `wasm32-wasip1-threads`. The need for this rename is a bit unfortunate as the previous name was chosen in an attempt to be future-compatible with other WASI targets. Originally this target was proposed to be `wasm32-wasi-threads`, and that's what was originally implemented in wasi-sdk as well. After discussion though and with the plans for the upcoming component-model target (now named `wasm32-wasip2`) the "preview1" naming was chosen for the threads-based target. The WASI subgroup later decided that it was time to drop the "preview" terminology and recommends "pX" instead, hence previous PRs to add `wasm32-wasip2` and rename `wasm32-wasi` to `wasm32-wasip1`.
So, with all that history, the "proper name" for this target is different than its current name, so one way or another a rename is required. This PR proposes renaming this target cold-turkey, unlike `wasm32-wasi` which is having a long transition period to change its name. The threads-based target is predicted to see only a fraction of the traffic of `wasm32-wasi` due to the unstable nature of the WASI threads proposal itself.
While I was here I updated the in-tree documentation in the target spec file itself as most of the documentation was copied from the original WASI target and wasn't as applicable to this target.
Also, as an aside, I can at least try to apologize for all the naming confusion here, but this is hopefully the last WASI-related rename.
Test wasm32-wasip1 in CI, not wasm32-unknown-unknown
This commit changes CI to no longer test the `wasm32-unknown-unknown` target and instead test the `wasm32-wasip1` target. There was some discussion of this in a [Zulip thread], and the motivations for this PR are:
* Runtime failures on `wasm32-unknown-unknown` print nothing, meaning all you get is "something failed". In contrast `wasm32-wasip1` can print to stdout/stderr.
* The unknown-unknown target is missing lots of pieces of libstd, and while `wasm32-wasip1` is also missing some pieces (e.g. threads) it's missing fewer pieces. This means that many more tests can be run.
Overall my hope is to improve the debuggability of wasm failures on CI and ideally be a bit less of a maintenance burden.
This commit specifically removes the testing of `wasm32-unknown-unknown` and replaces it with testing of `wasm32-wasip1`. Along the way there were a number of other archiectural changes made as well, including:
* A new `target.*.runtool` option can now be specified in `config.toml` which is passed as `--runtool` to `compiletest`. This is used to reimplement execution of WebAssembly in a less-wasm-specific fashion.
* The default value for `runtool` is an ambiently located WebAssembly runtime found on the system, if any. I've implemented logic for Wasmtime.
* Existing testing support for `wasm32-unknown-unknown` and Emscripten has been removed. I'm not aware of Emscripten testing being run any time recently and otherwise `wasm32-wasip1` is in theory the focus now.
* I've added a new `//@ needs-threads` directive for `compiletest` and classified a bunch of wasm-ignored tests as needing threads. In theory these tests can run on `wasm32-wasi-preview1-threads`, for example.
* I've tried to audit all existing tests that are either `ignore-emscripten` or `ignore-wasm*`. Many now run on `wasm32-wasip1` due to being able to emit error messages, for example. Many are updated with comments as to why they can't run as well.
* The `compiletest` output matching for `wasm32-wasip1` automatically uses "match a subset" mode implemented in `compiletest`. This is because WebAssembly runtimes often add extra information on failure, such as the `unreachable` instruction in `panic!`, which isn't able to be matched against the golden output from native platforms.
* I've ported most existing `run-make` tests that use custom Node.js wrapper scripts to the new run-make-based-in-Rust infrastructure. To do this I added `wasmparser` as a dependency of `run-make-support` for the various wasm tests to use that parse wasm files. The one test that executed WebAssembly now uses `wasmtime`-the-CLI to execute the test instead. I have not ported over an exception-handling test as Wasmtime doesn't implement this yet.
* I've updated the `test` crate to print out timing information for WASI targets as it can do that (gets a previously ignored test now passing).
* The `test-various` image now builds a WASI sysroot for the WASI target and additionally downloads a fixed release of Wasmtime, currently the latest one at 18.0.2, and uses that for testing.
[Zulip thread]: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Have.20wasm.20tests.20ever.20caused.20problems.20on.20CI.3F/near/424317944
This commit renames the current `wasm32-wasi-preview1-threads` target to
`wasm32-wasip1-threads`. The need for this rename is a bit unfortunate
as the previous name was chosen in an attempt to be future-compatible
with other WASI targets. Originally this target was proposed to be
`wasm32-wasi-threads`, and that's what was originally implemented in
wasi-sdk as well. After discussion though and with the plans for the
upcoming component-model target (now named `wasm32-wasip2`) the
"preview1" naming was chosen for the threads-based target. The WASI
subgroup later decided that it was time to drop the "preview"
terminology and recommends "pX" instead, hence previous PRs to add
`wasm32-wasip2` and rename `wasm32-wasi` to `wasm32-wasip1`.
So, with all that history, the "proper name" for this target is
different than its current name, so one way or another a rename is
required. This PR proposes renaming this target cold-turkey, unlike
`wasm32-wasi` which is having a long transition period to change its
name. The threads-based target is predicted to see only a fraction of
the traffic of `wasm32-wasi` due to the unstable nature of the WASI
threads proposal itself.
While I was here I updated the in-tree documentation in the target spec
file itself as most of the documentation was copied from the original
WASI target and wasn't as applicable to this target.
Also, as an aside, I can at least try to apologize for all the naming
confusion here, but this is hopefully the last WASI-related rename.
