cleanup: use run_in_tmpdir in run-make/rustdoc-scrape-examples-paths
We had an issue on our LLVM-head Rust builder where it got stuck with this test failing because it was reusing the tmpdir between runs and something broke the incremental compile. Everything seems to work fine with run_in_tmpdir in this test. tests/run-make/uefi-qemu also uses the same tmpdir across runs, but I don't have the right environment to test that so I didn't try fixing it. That is the only use of std::env::temp_dir left in run-make tests after this fix.
rustdoc-search: search backend with partitioned suffix tree
Before:
- https://notriddle.com/windows-docs-rs/doc-old/windows/
- https://doc.rust-lang.org/1.89.0/std/index.html
- https://doc.rust-lang.org/1.89.0/nightly-rustc/rustc_hir/index.html
After:
- https://notriddle.com/windows-docs-rs/doc/windows/
- https://notriddle.com/rustdoc-html-demo-12/stringdex/doc/std/index.html
- https://notriddle.com/rustdoc-html-demo-12/stringdex/compiler-doc/rustc_hir/index.html
## Summary
Rewrites the rustdoc search engine to use an indexed data structure, factored out as a crate called [stringdex](https://crates.io/crates/stringdex), that allows it to perform modified-levenshtein distance calculations, substring matches, and prefix matches in a reasonably efficient, and, more importantly, *incremental* algorithm.
## Motivation
Fixes https://github.com/rust-lang/rust/issues/131156
While the windows-rs crate is definitely the worst offender, I've noticed performance problems with the compiler crates as well. It makes no sense for rustdoc-search to have poor performance: it's basically a spell checker, and those have been usable since the 90's.
Stringdex is particularly designed to quickly return exact matches, to always report those matches first, and to never, ever [place new matches on top of old ones](https://web.dev/articles/cls). It also tries to yield to the event loop occasionally as it runs. This way, you can click the exactly-matched result before the rest of the search finishes running.
## Explanation
A longer description of how name search works can be found in stringdex's [HACKING.md](https://gitlab.com/notriddle/stringdex/-/blob/main/HACKING.md) file.
Type search is done by performing a name search on each element, then performing bitmap operations to narrow down a list of potential matches, then performing type unification on each match.
## Drawbacks
It's rather complex, and takes up more disk space than the current flat list of strings.
## Rationale and alternatives
Instead of a suffix tree, I could've used a different [approximate matching data structure](https://en.wikipedia.org/wiki/Approximate_string_matching). I didn't do that because I wanted to keep the current behavior (for example, a straightforward trigram index won't match [oepn](https://doc.rust-lang.org/nightly/std/?search=oepn) like the current system does).
## Prior art
[Sherlodoc](https://github.com/art-w/sherlodoc) is based on a similar concept, but they:
- use edit distance over a suffix tree for type-based search, instead of the binary matching that's implemented here
- use substring matching for name-based search, [but not fuzzy name matching](https://github.com/art-w/sherlodoc/issues/21)
- actually implement body text search, which is a potential-future feature, but not implemented in this PR
## Future possibilities
### Low-level optimization in stringdex
There are half a dozen low-level optimizations that I still need to explore. I haven't done them yet, because I've been working on bug fixes and rebasing on rustdoc's side, and a more solid and diverse test suite for stringdex itself.
- Stringdex decides whether to bundle two nodes into the same file based on size. To figure out a node's size, I have to run compression on it. This is probably slower than it needs to be.
- Stack compression is limited to the same 256-slot sliding windows as backref compression, and it doesn't have to be. (stack and backref compression are used to optimize the representation of the edge pointer from a parent node to its child; backref uses one byte, while stack is entirely implicit)
- The JS-side decoder is pretty naive. It performs unnecessary hash table lookups when decoding compressed nodes, and retains a list of hashes that it doesn't need. It needs to calculate the hashes in order to construct the merkle tree correctly, but it doesn't need to keep them.
- Data compression happens at the end, while emitting the node. This means it's not being counted when deciding on how to bundle, which is pretty dumb.
### Improved recall in type-driven search
Right now, type-driven search performs very strict matching. It's very precise, but misses a lot of things people would want.
What I'm not sure about is whether to focus more on edit-distance-based approaches, or to focus on type-theoretical approaches. Both gives avenues to improve, but edit distance is going to be faster while type checking is going to be more precise.
For example, a type theoretical improvement would fix [`Iterator<T>, (T -> U) -> Iterator<U>`](https://doc.rust-lang.org/nightly/std/?search=Iterator%3CT%3E%2C%20(T%20-%3E%20U)%20-%3E%20Iterator%3CU%3E) to give [`Iterator::map`](https://doc.rust-lang.org/nightly/std/iter/trait.Iterator.html#method.map), because it would recognize that the Map struct implements the Iterator trait. I don't know of any clean way to get this result to work without implementing significant type checking logic in search.js, and an edit-distance-based "dirty" approach would likely give a bunch of other results on top of this one.
## Full-text search
Once you've got this fuzzy dictionary matching to work, the logical next step is to implement some kind of information retrieval-based approach to phrase matching.
Like applying edit distance to types, phrase search gets you significantly better recall, but with a few major drawbacks:
- You have to pick between index bloat and the use of stopwords. Stopwords are bad because they might actually be important (try searching "if let" in mdBook if you're feeling brave), but without them, you spend a lot of space on text that doesn't matter.
- Example code also tends to have a lot of irrelevant stuff in it. Like stop words, we'd have to pick potentially-confusing or bloat.
Neither of these problems are deal-breakers, but they're worth keeping in mind.
library: Migrate from `cfg_if` to `cfg_select`
Migrate the standard library from using the external `cfg_if` crate to using the now-built-in `cfg_select` macro.
This does not yet eliminate the dependency from `library/std/Cargo.toml`, because while the standard library itself no longer uses `cfg_if`, it also incorporates the `backtrace` crate, which does.
Migration assisted by the following vim command (after selecting the full `cfg_if!` invocation):
```
'<,'>s/\(cfg_if::\)\?cfg_if/cfg_select/ | '<,'>s/^\( *\)} else {/\1}\r\1_ => {/c | '<,'>s/^\( *\)} else if #\[cfg(\(.*\))\] /\1}\r\1\2 => /e | '<,'>s/if #\[cfg(\(.*\))\] {/\1 => {/e
```
This is imperfect, but substantially accelerated the process. This prompts for confirmation on the `} else {` since that can also appear inside one of the arms. This also requires manual intervention to handle any multi-line conditions.
We had an issue on our LLVM-head Rust builder where it got stuck with
this test failing because it was reusing the tmpdir between runs and
something broke the incremental compile. Everything seems to work fine
with run_in_tmpdir in this test. tests/run-make/uefi-qemu also uses the
same tmpdir across runs, but I don't have the right environment to test
that so I didn't try fixing it. That is the only use of
std::env::temp_dir left in run-make tests after this fix.
strip prefix of temporary file names when it exceeds filesystem name length limit
When doing lto, rustc generates filenames that are concatenating many information.
In the case of this testcase, it is concatenating crate name and rust file name, plus some hash, and the extension. In some other cases it will concatenate even more information reducing the maximum effective crate name to about 110 chars on linux filesystems where filename max length is 255
This commit is ensuring that the temporary file names are limited in size, while still reasonably ensuring the unicity (with hashing of the stripped part)
Fix: rust-lang/rust#49914
Make no_mangle on foreign items explicit instead of implicit
for a followup PR I'm working on I need some foreign items to mangle. I could add a new attribute: `no_no_mangle` or something silly like that but by explicitly putting `no_mangle` in the codegen fn attrs of foreign items we can default it to `no_mangle` and then easily remove it when we don't want it.
I guess you'd know about this r? `@bjorn3.` Shouldn't be too hard to review :)
Builds on rust-lang/rust#144655 which should merge first.
The exact reasoning why we do not always pass the SDK root when linking
on macOS eludes me, but I suspect it's because we want to support
compiler drivers which do not support the `-isysroot` option.
Since we now pass the SDK root via the environment variable SDKROOT,
compiler drivers that don't support it can just ignore it.
Similarly, since we only warn when xcrun fails, users that expect their
compiler driver to provide the SDK location can do so now.
When doing lto, rustc generates filenames that are concatenating many information.
In the case of this testcase, it is concatenating crate name and rust file name, plus some hash, and the extension.
In some other cases it will concatenate even more information reducing the maximum effective crate name to about 110 chars on linux filesystems where
filename max length is 255
This commit is ensuring that the temporary file names are limited in size, while still reasonabily ensuring the unicity (with hashing of the stripped part)
[win][arm64ec] Partial fix for raw-dylib-link-ordinal on Arm64EC
These are the test fixes required to get `raw-dylib-link-ordinal` working on Arm64EC.
For the test to completely pass, we also need an updated `ar_archive_writer` with <https://github.com/rust-lang/ar_archive_writer/pull/24> merged in.
atomicrmw on pointers: move integer-pointer cast hacks into backend
Conceptually, we want to have atomic operations on pointers of the form `fn atomic_add(ptr: *mut T, offset: usize, ...)`. However, LLVM does not directly support such operations (https://github.com/llvm/llvm-project/issues/120837), so we have to cast the `offset` to a pointer somewhere.
This PR moves that hack into the LLVM backend, so that the standard library, intrinsic, and Miri all work with the conceptual operation we actually want. Hopefully, one day LLVM will gain a way to represent these operations without integer-pointer casts, and then the hack will disappear entirely.
Cc ```@nikic``` -- this is the best we can do right now, right?
Fixes https://github.com/rust-lang/rust/issues/134617
[win][arm64ec] Fix msvc-wholearchive for Arm64EC
`msvc-wholearchive` was failing on Arm64EC Windows as it requires the `/MACHINE:ARM64EC` flag to be passed to the MSVC linker.
This required splitting the `extra_c_flags` function into a new `extra_linker_flags` function as `/MACHINE:ARM64EC` is not a valid argument to be passed to the MSVC Compiler (instead, `/arm64EC` should be used).
bump bootstrap compiler to 1.90 beta
There were significantly less `cfg(bootstrap)` and `cfg(not(bootstrap))` this release. Presumably due to the fact that we change the bootstrap stage orderings to reduce the need for them and it was successful 🙏
Revert "Preserve the .debug_gdb_scripts section"
https://github.com/rust-lang/rust/pull/143679 introduces a significant build time perf regression for ripgrep. Let's revert it such that we can investigate it without pressure.
Print thread ID in panic message
`panic!` does not print any identifying information for threads that are
unnamed. However, in many cases, the thread ID can be determined.
This changes the panic message from something like this:
thread '<unnamed>' panicked at src/main.rs:3:5:
explicit panic
To something like this:
thread '<unnamed>' (12345) panicked at src/main.rs:3:5:
explicit panic
Stack overflow messages are updated as well.
This change applies to both named and unnamed threads. The ID printed is
the OS integer thread ID rather than the Rust thread ID, which should
also be what debuggers print.
try-job: aarch64-apple
try-job: aarch64-gnu
try-job: dist-apple-various
try-job: dist-various-*
try-job: dist-x86_64-freebsd
try-job: dist-x86_64-illumos
try-job: dist-x86_64-netbsd
try-job: dist-x86_64-solaris
try-job: test-various
try-job: x86_64-gnu
try-job: x86_64-mingw-1
try-job: x86_64-msvc-1
`panic!` does not print any identifying information for threads that are
unnamed. However, in many cases, the thread ID can be determined.
This changes the panic message from something like this:
thread '<unnamed>' panicked at src/main.rs:3:5:
explicit panic
To something like this:
thread '<unnamed>' (0xff9bf) panicked at src/main.rs:3:5:
explicit panic
Stack overflow messages are updated as well.
This change applies to both named and unnamed threads. The ID printed is
the OS integer thread ID rather than the Rust thread ID, which should
also be what debuggers print.
Fix rustdoc scrape examples crash
Fixesrust-lang/rust#144752.
The regression was introduced in https://github.com/rust-lang/rust/pull/144600. Although I don't understand why it is an issue currently, this allows to bypass the failure for now until we can figure out what's wrong as it's currently blocking new `bevy`'s release.
cc `@alice-i-cecile`
r? `@fmease`
Make sure that compiler and linker don't optimize the section's contents
away by adding the global holding the data to "llvm.used". The volatile
load in the main shim is retained because "llvm.used", which translates
to SHF_GNU_RETAIN on ELF targets, requires a reasonably recent linker;
emitting the volatile load ensures compatibility with older linkers, at
least when libstd is used.
Pretty printers in dylib dependencies are now emitted by the main crate
instead of the dylib; apart from matching how rlibs are handled, this
approach has the advantage that `omit_gdb_pretty_printer_section` keeps
working with dylib dependencies.
update fortanix tests
Firstly, as far as I can tell, no CI job actually runs any of the fortanix tests? Maybe I'm missing the job that runs these tests though?
In any case, the `assembly` tests now use `minicore`, meaning that they will run regardless of the host architecture (specifically, they will run during a standard PR CI build).
The run-make test is actually broken, and I'd propose to make it just `cargo build` rather than `cargo run`. We can have a separate test for actually running the program, if desired.
Also this test is subject to https://github.com/rust-lang/rust/issues/128733, so I'd like to re-evaluate what parts of the C/C++ compilation are actually required or useful.
cc [``@jethrogb](https://github.com/jethrogb)`` [``@raoulstrackx](https://github.com/raoulstrackx)`` [``@aditijannu](https://github.com/aditijannu)``
r? ``@jieyouxu``
Fix linker-plugin-lto only doing thin lto
When rust provides LLVM bitcode files to lld and the bitcode contains
function summaries as used for thin lto, lld defaults to using thin lto.
This prevents some optimizations that are only applied for fat lto.
We solve this by not creating function summaries when fat lto is
enabled. The bitcode for the module is just directly written out.
An alternative solution would be to set the `ThinLTO=0` module flag to
signal lld to do fat lto.
The code in clang that sets this flag is here:
560149b5e3/clang/lib/CodeGen/BackendUtil.cpp (L1150)
The code in LLVM that queries the flag and defaults to thin lto if not
set is here:
e258bca950/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp (L4441-L4446)
try-job: x86_64-gnu-debug
try-job: aarch64-gnu-debug
When rust provides LLVM bitcode files to lld and the bitcode contains
function summaries as used for thin lto, lld defaults to using thin lto.
This prevents some optimizations that are only applied for fat lto.
We solve this by not creating function summaries when fat lto is
enabled. The bitcode for the module is just directly written out.
An alternative solution would be to set the `ThinLTO=0` module flag to
signal lld to do fat lto.
The code in clang that sets this flag is here:
560149b5e3/clang/lib/CodeGen/BackendUtil.cpp (L1150)
The code in LLVM that queries the flag and defaults to thin lto if not
set is here:
e258bca950/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp (L4441-L4446)
feat: Right align line numbers
As part of my work on getting `annotate-snipptes` to be used as `rustc`'s renderer, I realized that `rustc` left-aligned line numbers, while `annotate-snippets` right-aligned them. This PR switches `rustc` to right-align the line numbers, matching `annotate-snippets`. In practice, this change isn't very noticeable in day-to-day output, as it only shows up when a diagnostic span contains line numbers with different lengths (9->10, 99->100, 999->1000, etc.).
`rustc`
```
error[E0412]: cannot find type `F` in this scope
--> $DIR/ui-testing-optout.rs:92:10
|
4 | type A = B;
| ----------- similarly named type alias `A` defined here
...
92 | type E = F;
| ^ help: a type alias with a similar name exists: `A`
```
`annotate-snippets`
```
error[E0412]: cannot find type `F` in this scope
--> $DIR/ui-testing-optout.rs:92:10
|
4 | type A = B;
| ----------- similarly named type alias `A` defined here
...
92 | type E = F;
| ^ help: a type alias with a similar name exists: `A`
```
r? ``@compiler-errors``
Ensure external paths passed via flags end up in rustdoc depinfo
rustdoc has many flags to pass external HTML/Markdown/CSS files that end up in the build. These need to be recorded in depinfo so that Cargo will rebuild the crate if they change.
rustdoc has many flags to pass external HTML/Markdown/CSS files that end
up in the build. These need to be recorded in depinfo so that Cargo will
rebuild the crate if they change.
rustdoc: save target modifiers
`rustdoc` was filling a `target_modifiers` variable, but it was not using the result.
In turn, that means that trying to use a dependency that set a target modifier fails.
For instance, running:
```sh
RUSTC_BOOTSTRAP=1 rustc --edition=2024 --target=aarch64-unknown-none-softfloat --sysroot=/dev/null --emit=metadata -Zfixed-x18 --crate-type rlib --crate-name core $(rustc --print sysroot)/lib/rustlib/src/rust/library/core/src/lib.rs
echo '#![allow(internal_features)]
' | RUSTC_BOOTSTRAP=1 rustdoc --edition=2021 --target=aarch64-unknown-none-softfloat --sysroot=/dev/null -Zfixed-x18 --extern core=libcore.rmeta -
```
will fail with:
```text
error: mixing `-Zfixed-x18` will cause an ABI mismatch in crate `rust_out`
|
= help: the `-Zfixed-x18` flag modifies the ABI so Rust crates compiled with different values of this flag cannot be used together safely
= note: unset `-Zfixed-x18` in this crate is incompatible with `-Zfixed-x18=` in dependency `core`
= help: set `-Zfixed-x18=` in this crate or unset `-Zfixed-x18` in `core`
= help: if you are sure this will not cause problems, you may use `-Cunsafe-allow-abi-mismatch=fixed-x18` to silence this error
```
Thus save the targets modifiers in `Options` to then pass it to the session options, so that eventually the diff can be performed as expected in `report_incompatible_target_modifiers()`.
Cc: ``@azhogin``
Fixes: https://github.com/rust-lang/rust/issues/144521
move uefi test to run-make
Turn the `uefi` test into a more standard `run-make` test, and execute it using the `test-various` CI job like before.
This is just a straightforward translation of the python code, but using `run-make` to supply the target (hence the 3 separate calls in the docker file).
r? ```@jieyouxu```
cc ```@nicholasbishop```
try-job: test-various
`rustdoc` was filling a `target_modifiers` variable, but it was not
using the result.
In turn, that means that trying to use a dependency that set a target
modifier fails.
For instance, running:
```sh
RUSTC_BOOTSTRAP=1 rustc --edition=2024 --target=aarch64-unknown-none-softfloat --sysroot=/dev/null --emit=metadata -Zfixed-x18 --crate-type rlib --crate-name core $(rustc --print sysroot)/lib/rustlib/src/rust/library/core/src/lib.rs
echo '#![allow(internal_features)]
' | RUSTC_BOOTSTRAP=1 rustdoc --edition=2021 --target=aarch64-unknown-none-softfloat --sysroot=/dev/null -Zfixed-x18 --extern core=libcore.rmeta -
```
will fail with:
```text
error: mixing `-Zfixed-x18` will cause an ABI mismatch in crate `rust_out`
|
= help: the `-Zfixed-x18` flag modifies the ABI so Rust crates compiled with different values of this flag cannot be used together safely
= note: unset `-Zfixed-x18` in this crate is incompatible with `-Zfixed-x18=` in dependency `core`
= help: set `-Zfixed-x18=` in this crate or unset `-Zfixed-x18` in `core`
= help: if you are sure this will not cause problems, you may use `-Cunsafe-allow-abi-mismatch=fixed-x18` to silence this error
```
Thus save the targets modifiers in `Options` to then pass it to the
session options, so that eventually the diff can be performed as expected
in `report_incompatible_target_modifiers()`.
Fixes: https://github.com/rust-lang/rust/issues/144521
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Use serde for target spec json deserialize
The previous manual parsing of `serde_json::Value` was a lot of complicated code and extremely error-prone. It was full of janky behavior like sometimes ignoring type errors, sometimes erroring for type errors, sometimes warning for type errors, and sometimes just ICEing for type errors (the icing on the top).
Additionally, many of the error messages about allowed values were out of date because they were in a completely different place than the FromStr impls. Overall, the system caused confusion for users.
I also found the old deserialization code annoying to read. Whenever a `key!` invocation was found, one had to first look for the right macro arm, and no go to definition could help.
This PR replaces all this manual parsing with a 2-step process involving serde.
First, the string is parsed into a `TargetSpecJson` struct. This struct is a 1:1 representation of the spec JSON. It already parses all the enums and is very simple to read and write.
Then, the fields from this struct are copied into the actual `Target`. The reason for this two-step process instead of just serializing into a `Target` is because of a few reasons
1. There are a few transformations performed between the two formats
2. The default logic is implemented this way. Otherwise all the default field values would have to be spelled out again, which is suboptimal. With this logic, they fall out naturally, because everything in the json struct is an `Option`.
Overall, the mapping is pretty simple, with the vast majority of fields just doing a 1:1 mapping that is captured by two macros. I have deliberately avoided making the macros generic to keep them simple.
All the `FromStr` impls now have the error message right inside them, which increases the chance of it being up to date. Some "`from_str`" impls were turned into proper `FromStr` impls to support this.
The new code is much less involved, delegating all the JSON parsing logic to serde, without any manual type matching.
This change introduces a few breaking changes for consumers. While it is possible to use this format on stable, it is very much subject to change, so breaking changes are expected. The hope is also that because of the way stricter behavior, breaking changes are easier to deal with, as they come with clearer error messages.
1. Invalid types now always error, everywhere. Previously, they would sometimes error, and sometimes just be ignored (which meant the users JSON was still broken, just silently!)
2. This now makes use of `deny_unknown_fields` instead of just warning on unused fields, which was done previously. Serde doesn't make it easy to get such warning behavior, which was the primary reason that this now changed. But I think error behavior is very reasonable too. If someone has random stale fields in their JSON, it is likely because these fields did something at some point but no longer do, and the user likely wants to be informed of this so they can figure out what to do.
This is also relevant for the future. If we remove a field but someone has it set, it probably makes sense for them to take a look whether they need this and should look for alternatives, or whether they can just delete it. Overall, the JSON is made more explicit.
This is the only expected breakage, but there could also be small breakage from small mistakes. All targets roundtrip though, so it can't be anything too major.
fixesrust-lang/rust#144153
The previous manual parsing of `serde_json::Value` was a lot of
complicated code and extremely error-prone. It was full of janky
behavior like sometimes ignoring type errors, sometimes erroring for
type errors, sometimes warning for type errors, and sometimes just
ICEing for type errors (the icing on the top).
Additionally, many of the error messages about allowed values were out
of date because they were in a completely different place than the
FromStr impls. Overall, the system caused confusion for users.
I also found the old deserialization code annoying to read. Whenever a
`key!` invocation was found, one had to first look for the right macro
arm, and no go to definition could help.
This PR replaces all this manual parsing with a 2-step process involving
serde.
First, the string is parsed into a `TargetSpecJson` struct. This struct
is a 1:1 representation of the spec JSON. It already parses all the
enums and is very simple to read and write.
Then, the fields from this struct are copied into the actual `Target`.
The reason for this two-step process instead of just serializing into a
`Target` is because of a few reasons
1. There are a few transformations performed between the two formats
2. The default logic is implemented this way. Otherwise all the default
field values would have to be spelled out again, which is
suboptimal. With this logic, they fall out naturally, because
everything in the json struct is an `Option`.
Overall, the mapping is pretty simple, with the vast majority of fields
just doing a 1:1 mapping that is captured by two macros. I have
deliberately avoided making the macros generic to keep them simple.
All the `FromStr` impls now have the error message right inside them,
which increases the chance of it being up to date. Some "`from_str`"
impls were turned into proper `FromStr` impls to support this.
The new code is much less involved, delegating all the JSON parsing
logic to serde, without any manual type matching.
This change introduces a few breaking changes for consumers. While it is
possible to use this format on stable, it is very much subject to
change, so breaking changes are expected. The hope is also that because
of the way stricter behavior, breaking changes are easier to deal with,
as they come with clearer error messages.
1. Invalid types now always error, everywhere. Previously, they would
sometimes error, and sometimes just be ignored (which meant the users
JSON was still broken, just silently!)
2. This now makes use of `deny_unknown_fields` instead of just warning
on unused fields, which was done previously. Serde doesn't make it
easy to get such warning behavior, which was the primary reason that
this now changed. But I think error behavior is very reasonable too.
If someone has random stale fields in their JSON, it is likely
because these fields did something at some point but no longer do,
and the user likely wants to be informed of this so they can figure
out what to do.
This is also relevant for the future. If we remove a field but
someone has it set, it probably makes sense for them to take a look
whether they need this and should look for alternatives, or whether
they can just delete it. Overall, the JSON is made more explicit.
This is the only expected breakage, but there could also be small
breakage from small mistakes. All targets roundtrip though, so it can't
be anything too major.
