tests/run-make/crate-loading: Rename source files for clarity
For rust-lang/rust#146874 I originally tried to extend the existing test **tests/run-make/crate-loading**. That didn't work out since adding a re-export of the entire crate significantly changes the emitted error messsage.
I did put some effort into making that test easier to understand however, by renaming its files. (Since I was confused myself at first.) Let's save some time for future devs by doing just the renames.
Further cleanups are possible, but that will change the blessed output, so let's not do that right now.
r? ```@jieyouxu``` since you have the context of rust-lang/rust#146874
fix a crash in rustdoc merge finalize without input file
- Closesrust-lang/rust#146646
`SerializedSearchIndex::union` calls `Symbol::intern`, requiring `SESSION_GLOBALS` to be set.
I missed this target when I changed all the other tier 3 targets. Only
realized that this one was still statically linked when I looked at the
list of targets in the test later.
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
All of the tier 3 targets in the list now link dynamically by default
(except mips64el-unknown-linux-muslabi64, I overlooked that one).
Adjust the list of targets expected to link statically accordingly.
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
This schema is helpful for people writing custom target spec JSON. It
can provide autocomplete in the editor, and also serves as documentation
when there are documentation comments on the structs, as `schemars` will
put them in the schema.
Split `run-make` into two {`run-make`,`run-make-cargo`} test suites
## Summary
Split `tests/run-make` into two test suites, to make it faster and more convenient for contributors to run run-make tests that do not need in-tree `cargo`.
| New test suites | Explanation |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tests/run-make` | The "fast path" test suite intended for run-make tests that do not need in-tree `cargo`. These tests may not use `cargo`. |
| `tests/run-make-cargo` | The "slow path" test suite that requires checking out `cargo` submodule and building in-tree `cargo`, and thus will have access to in-tree `cargo`. In practice, these constitute a very small portion of the original `run-make` tests. |
This PR carries out [MCP 847: Split run-make test suite into slower-building test suite with suitably-staged cargo and faster-building test suite without cargo](https://github.com/rust-lang/compiler-team/issues/847).
Fixesrust-lang/rust#135573 (for the tests that do not need in-tree `cargo`).
Fixesrust-lang/rust#134109.
## Remarks
- I considered if we want to split by in-tree tools previously. However, as discussed rust-lang/rust#134109, in practice `rustdoc` is not very slow to build, but `cargo` takes a good few minutes. So, the partition boundary was determined to be along in-tree `cargo` availability.
- The `run-make` tests previously that wanted to use `cargo` cannot just use the bootstrap `cargo`, otherwise they would run into situations where bootstrap `cargo` can significantly diverge from in-tree `cargo` (see https://github.com/rust-lang/rust/pull/130642).
---
try-job: aarch64-msvc-1
try-job: test-various
try-job: x86_64-gnu-debug
try-job: aarch64-gnu-debug
try-job: aarch64-apple
try-job: dist-various-1
Disallow shebang in `--cfg` and `--check-cfg` arguments
This PR is similar to https://github.com/rust-lang/rust/issues/146130, where we disallowed frontmatter in `--cfg` and `--check-cfg` arguments. While fixing the other one we also discovered that shebang `#!/usr/bin/shebang` are currently also allowed in `--cfg` and `--check-cfg` arguments.
Allowing shebang in them (which are just ignored) was never intended, this PR fixes that by not stripping shebang for `--cfg` and `--check-cfg` arguments.
This is technically a breaking-change, although I don't expect anyone to actually rely on this unintended behavior.
Fixes https://github.com/rust-lang/rust/issues/146130#issuecomment-3246299499
r? fmease
Add `__isPlatformVersionAtLeast` and `__isOSVersionAtLeast` symbols
## Motivation
When Objective-C code uses ```@available(...)`,`` Clang inserts a call to [`__isPlatformVersionAtLeast`](https://github.com/llvm/llvm-project/blob/llvmorg-20.1.0/compiler-rt/lib/builtins/os_version_check.c#L276) (`__isOSVersionAtLeast` in older Clang versions). These symbols not being available sometimes ends up causing linker errors. See the new test `tests/run-make/apple-c-available-links` for a minimal reproducer.
The workaround is to link `libclang_rt.osx.a`, see e.g. https://github.com/alexcrichton/curl-rust/issues/279. But that's very difficult for users to figure out (and the backreferences to that issue indicates that people are still running into this in their own projects every so often).
For another recent example, this is preventing `rustc` from using LLVM assertions on macOS, see https://github.com/rust-lang/rust/pull/62592#issuecomment-510670657 and https://github.com/rust-lang/rust/pull/134275#issuecomment-2543067830.
It is also a blocker for [setting the correct minimum OS version in `cc-rs`](https://github.com/rust-lang/rust/issues/136113), since fixing this in `cc-rs` might end up introducing linker errors in places where we weren't before (by default, if using e.g. ```@available(macos`` 10.15, *)`, the symbol usually happens to be left out, since `clang` defaults to compiling for the host macOS version, and thus things _seem_ to work - but the availability check actually compiles down to nothing, which is a huge correctness footgun for running on older OSes).
(My super secret evil agenda is also to expose some variant of ```@available``` in Rust's `std` after https://github.com/rust-lang/rfcs/pull/3750 progresses further, will probably file an ACP for this later. But I believe this PR has value regardless of those future plans, since we'd be making C/Objective-C/Swift interop easier).
## Solution
Implement `__isPlatformVersionAtLeast` and `__isOSVersionAtLeast` as part of the "public ABI" that `std` exposes.
**This is insta-stable**, in the same sense that additions to `compiler-builtins` are insta-stable, though the availability of these symbols can probably be considered a "quality of implementation" detail rather than a stable promise.
I originally proposed to implement this in `compiler-builtins`, see https://github.com/rust-lang/compiler-builtins/pull/794, but we discussed moving it to `std` instead ([Zulip thread](https://rust-lang.zulipchat.com/#narrow/channel/219381-t-libs/topic/Provide.20.60__isPlatformVersionAtLeast.60.20in.20.60std.60.3F/with/507880717)), which makes the implementation substantially simpler, and we avoid gnarly issues with requiring the user to link `libSystem.dylib` (since `std` unconditionally does that).
Note that this does not solve the linker errors for (pure) `#![no_std]` users, but that's _probably_ fine, if you are using ```@available``` to test the OS version on Apple platforms, you're likely also using `std` (and it is still possible to work around by linking `libclang_rt.*.a`).
A thing to note about the implementation, I've choosen to stray a bit from LLVM's upstream implementation, and not use `_availability_version_check` since [it has problems when compiling with an older SDK](https://github.com/llvm/llvm-project/issues/64227). Instead, we use `sysctl kern.osproductversion` when available to still avoid the costly PList lookup in most cases, but still with a fall back to the PList lookup when that is not available (with the PList fallback being is similar to LLVM's implementation).
## Testing
Apple has a lot of different "modes" that they can run binaries in, which can be a bit difficult to find your bearings in, but I've tried to be as thorough as I could in testing them all.
Tested using roughly the equivalent of `./x test library/std -- platform_version` on the following configurations:
- macOS 14.7.3 on a Macbook Pro M2
- `aarch64-apple-darwin`
- `x86_64-apple-darwin` (under Rosetta)
- `aarch64-apple-ios-macabi`
- `x86_64-apple-ios-macabi` (under Rosetta)
- `aarch64-apple-ios` (using Xcode's "Designed for iPad" setting)
- `aarch64-apple-ios-sim` (in iOS Simulator, as iPhone with iOS 17.5)
- `aarch64-apple-ios-sim` (in iOS Simulator, as iPad with iOS 18.2)
- `aarch64-apple-tvos-sim` (in tvOS Simulator)
- `aarch64-apple-watchos-sim` (in watchOS Simulator)
- `aarch64-apple-ios-sim` (in visionOS simulator, using Xcode's "Designed for iPad" setting)
- `aarch64-apple-visionos-sim` (in visionOS Simulator)
- macOS 15.3.1 VM
- `aarch64-apple-darwin`
- `aarch64-apple-ios-macabi`
- macOS 10.12.6 on an Intel Macbook from 2013
- `x86_64-apple-darwin`
- `i686-apple-darwin`
- `x86_64-apple-ios` (in iOS Simulator)
- iOS 9.3.6 on a 1st generation iPad Mini
- `armv7-apple-ios` with an older compiler
Along with manually inspecting the output of `version_from_sysctl()` and `version_from_plist()`, and verifying that they actually match what's expected.
I believe the only real omissions here would be:
- `aarch64-apple-ios` on a newer iPhone that has `sysctl` available (iOS 11.4 or above).
- `aarch64-apple-ios` on a Vision Pro using Xcode's "Designed for iPad" setting.
But I don't have the hardware available to test those.
``@rustbot`` label O-apple A-linkage -T-compiler -A-meta -A-run-make
try-job: aarch64-apple
Sanitizers target modificators
Depends on bool flag fix: https://github.com/rust-lang/rust/pull/138483.
Some sanitizers need to be target modifiers, and some do not. For now, we should mark all sanitizers as target modifiers except for these: AddressSanitizer, LeakSanitizer
For kCFI, the helper flag -Zsanitizer-cfi-normalize-integers should also be a target modifier.
Many test errors was with sanizer flags inconsistent with std deps. Tests are fixed with `-C unsafe-allow-abi-mismatch`.
fix(rustdoc): match rustc `--emit` precedence
Resolvesrust-lang/rust#141664
This changes rustdoc's `--emit` to allow only one instance of each type, regardless of the actual data that `--emit` carries. This matches rustc's `--emit` behavior.
As of the writing, only `dep-info` emit type carries extra data.
Disallow frontmatter in `--cfg` and `--check-cfg` arguments
This PR disallows the frontmatter syntax in `--cfg` and `--check-cfg` arguments.
Fixes https://github.com/rust-lang/rust/issues/146130
r? fmease
a more general version of https://github.com/rust-lang/rust/pull/146080.
after a bit of hacking in [`fluent.rs`](https://github.com/rust-lang/rust/blob/master/compiler/rustc_fluent_macro/src/fluent.rs), i discovered that i'm not the only one that is bad at following guidelines 😅. this pr lowercases the first letter of all the error messages in the codebase.
(i did not change things that are traditionally uppercased such as _MIR_, _ABI_ or _C_)
i think it's reasonable to run a `@bors try` so all the test suite is checked, as i cannot run some of the tests on my machine. i double checked (and replaced manually) all the old error messages, but better be safe than sorry.
in the future i will try to add a check in `x test tidy` that errors if an error message starts with an uppercase letter.
Change rustdoc's `--emit` to allow only one instance of each type,
regardless of the actual data that `--emit` carries.
This matches rustc's `--emit` behavior.
As of the writing, only `dep-info` emit type carries extra data.
See <https://github.com/rust-lang/rust/issues/141664>
cleanup: use run_in_tmpdir in run-make/rustdoc-scrape-examples-paths
We had an issue on our LLVM-head Rust builder where it got stuck with this test failing because it was reusing the tmpdir between runs and something broke the incremental compile. Everything seems to work fine with run_in_tmpdir in this test. tests/run-make/uefi-qemu also uses the same tmpdir across runs, but I don't have the right environment to test that so I didn't try fixing it. That is the only use of std::env::temp_dir left in run-make tests after this fix.
rustdoc-search: search backend with partitioned suffix tree
Before:
- https://notriddle.com/windows-docs-rs/doc-old/windows/
- https://doc.rust-lang.org/1.89.0/std/index.html
- https://doc.rust-lang.org/1.89.0/nightly-rustc/rustc_hir/index.html
After:
- https://notriddle.com/windows-docs-rs/doc/windows/
- https://notriddle.com/rustdoc-html-demo-12/stringdex/doc/std/index.html
- https://notriddle.com/rustdoc-html-demo-12/stringdex/compiler-doc/rustc_hir/index.html
## Summary
Rewrites the rustdoc search engine to use an indexed data structure, factored out as a crate called [stringdex](https://crates.io/crates/stringdex), that allows it to perform modified-levenshtein distance calculations, substring matches, and prefix matches in a reasonably efficient, and, more importantly, *incremental* algorithm.
## Motivation
Fixes https://github.com/rust-lang/rust/issues/131156
While the windows-rs crate is definitely the worst offender, I've noticed performance problems with the compiler crates as well. It makes no sense for rustdoc-search to have poor performance: it's basically a spell checker, and those have been usable since the 90's.
Stringdex is particularly designed to quickly return exact matches, to always report those matches first, and to never, ever [place new matches on top of old ones](https://web.dev/articles/cls). It also tries to yield to the event loop occasionally as it runs. This way, you can click the exactly-matched result before the rest of the search finishes running.
## Explanation
A longer description of how name search works can be found in stringdex's [HACKING.md](https://gitlab.com/notriddle/stringdex/-/blob/main/HACKING.md) file.
Type search is done by performing a name search on each element, then performing bitmap operations to narrow down a list of potential matches, then performing type unification on each match.
## Drawbacks
It's rather complex, and takes up more disk space than the current flat list of strings.
## Rationale and alternatives
Instead of a suffix tree, I could've used a different [approximate matching data structure](https://en.wikipedia.org/wiki/Approximate_string_matching). I didn't do that because I wanted to keep the current behavior (for example, a straightforward trigram index won't match [oepn](https://doc.rust-lang.org/nightly/std/?search=oepn) like the current system does).
## Prior art
[Sherlodoc](https://github.com/art-w/sherlodoc) is based on a similar concept, but they:
- use edit distance over a suffix tree for type-based search, instead of the binary matching that's implemented here
- use substring matching for name-based search, [but not fuzzy name matching](https://github.com/art-w/sherlodoc/issues/21)
- actually implement body text search, which is a potential-future feature, but not implemented in this PR
## Future possibilities
### Low-level optimization in stringdex
There are half a dozen low-level optimizations that I still need to explore. I haven't done them yet, because I've been working on bug fixes and rebasing on rustdoc's side, and a more solid and diverse test suite for stringdex itself.
- Stringdex decides whether to bundle two nodes into the same file based on size. To figure out a node's size, I have to run compression on it. This is probably slower than it needs to be.
- Stack compression is limited to the same 256-slot sliding windows as backref compression, and it doesn't have to be. (stack and backref compression are used to optimize the representation of the edge pointer from a parent node to its child; backref uses one byte, while stack is entirely implicit)
- The JS-side decoder is pretty naive. It performs unnecessary hash table lookups when decoding compressed nodes, and retains a list of hashes that it doesn't need. It needs to calculate the hashes in order to construct the merkle tree correctly, but it doesn't need to keep them.
- Data compression happens at the end, while emitting the node. This means it's not being counted when deciding on how to bundle, which is pretty dumb.
### Improved recall in type-driven search
Right now, type-driven search performs very strict matching. It's very precise, but misses a lot of things people would want.
What I'm not sure about is whether to focus more on edit-distance-based approaches, or to focus on type-theoretical approaches. Both gives avenues to improve, but edit distance is going to be faster while type checking is going to be more precise.
For example, a type theoretical improvement would fix [`Iterator<T>, (T -> U) -> Iterator<U>`](https://doc.rust-lang.org/nightly/std/?search=Iterator%3CT%3E%2C%20(T%20-%3E%20U)%20-%3E%20Iterator%3CU%3E) to give [`Iterator::map`](https://doc.rust-lang.org/nightly/std/iter/trait.Iterator.html#method.map), because it would recognize that the Map struct implements the Iterator trait. I don't know of any clean way to get this result to work without implementing significant type checking logic in search.js, and an edit-distance-based "dirty" approach would likely give a bunch of other results on top of this one.
## Full-text search
Once you've got this fuzzy dictionary matching to work, the logical next step is to implement some kind of information retrieval-based approach to phrase matching.
Like applying edit distance to types, phrase search gets you significantly better recall, but with a few major drawbacks:
- You have to pick between index bloat and the use of stopwords. Stopwords are bad because they might actually be important (try searching "if let" in mdBook if you're feeling brave), but without them, you spend a lot of space on text that doesn't matter.
- Example code also tends to have a lot of irrelevant stuff in it. Like stop words, we'd have to pick potentially-confusing or bloat.
Neither of these problems are deal-breakers, but they're worth keeping in mind.
library: Migrate from `cfg_if` to `cfg_select`
Migrate the standard library from using the external `cfg_if` crate to using the now-built-in `cfg_select` macro.
This does not yet eliminate the dependency from `library/std/Cargo.toml`, because while the standard library itself no longer uses `cfg_if`, it also incorporates the `backtrace` crate, which does.
Migration assisted by the following vim command (after selecting the full `cfg_if!` invocation):
```
'<,'>s/\(cfg_if::\)\?cfg_if/cfg_select/ | '<,'>s/^\( *\)} else {/\1}\r\1_ => {/c | '<,'>s/^\( *\)} else if #\[cfg(\(.*\))\] /\1}\r\1\2 => /e | '<,'>s/if #\[cfg(\(.*\))\] {/\1 => {/e
```
This is imperfect, but substantially accelerated the process. This prompts for confirmation on the `} else {` since that can also appear inside one of the arms. This also requires manual intervention to handle any multi-line conditions.
We had an issue on our LLVM-head Rust builder where it got stuck with
this test failing because it was reusing the tmpdir between runs and
something broke the incremental compile. Everything seems to work fine
with run_in_tmpdir in this test. tests/run-make/uefi-qemu also uses the
same tmpdir across runs, but I don't have the right environment to test
that so I didn't try fixing it. That is the only use of
std::env::temp_dir left in run-make tests after this fix.
strip prefix of temporary file names when it exceeds filesystem name length limit
When doing lto, rustc generates filenames that are concatenating many information.
In the case of this testcase, it is concatenating crate name and rust file name, plus some hash, and the extension. In some other cases it will concatenate even more information reducing the maximum effective crate name to about 110 chars on linux filesystems where filename max length is 255
This commit is ensuring that the temporary file names are limited in size, while still reasonably ensuring the unicity (with hashing of the stripped part)
Fix: rust-lang/rust#49914
Make no_mangle on foreign items explicit instead of implicit
for a followup PR I'm working on I need some foreign items to mangle. I could add a new attribute: `no_no_mangle` or something silly like that but by explicitly putting `no_mangle` in the codegen fn attrs of foreign items we can default it to `no_mangle` and then easily remove it when we don't want it.
I guess you'd know about this r? `@bjorn3.` Shouldn't be too hard to review :)
Builds on rust-lang/rust#144655 which should merge first.
The exact reasoning why we do not always pass the SDK root when linking
on macOS eludes me, but I suspect it's because we want to support
compiler drivers which do not support the `-isysroot` option.
Since we now pass the SDK root via the environment variable SDKROOT,
compiler drivers that don't support it can just ignore it.
Similarly, since we only warn when xcrun fails, users that expect their
compiler driver to provide the SDK location can do so now.
When doing lto, rustc generates filenames that are concatenating many information.
In the case of this testcase, it is concatenating crate name and rust file name, plus some hash, and the extension.
In some other cases it will concatenate even more information reducing the maximum effective crate name to about 110 chars on linux filesystems where
filename max length is 255
This commit is ensuring that the temporary file names are limited in size, while still reasonabily ensuring the unicity (with hashing of the stripped part)
[win][arm64ec] Partial fix for raw-dylib-link-ordinal on Arm64EC
These are the test fixes required to get `raw-dylib-link-ordinal` working on Arm64EC.
For the test to completely pass, we also need an updated `ar_archive_writer` with <https://github.com/rust-lang/ar_archive_writer/pull/24> merged in.
atomicrmw on pointers: move integer-pointer cast hacks into backend
Conceptually, we want to have atomic operations on pointers of the form `fn atomic_add(ptr: *mut T, offset: usize, ...)`. However, LLVM does not directly support such operations (https://github.com/llvm/llvm-project/issues/120837), so we have to cast the `offset` to a pointer somewhere.
This PR moves that hack into the LLVM backend, so that the standard library, intrinsic, and Miri all work with the conceptual operation we actually want. Hopefully, one day LLVM will gain a way to represent these operations without integer-pointer casts, and then the hack will disappear entirely.
Cc ```@nikic``` -- this is the best we can do right now, right?
Fixes https://github.com/rust-lang/rust/issues/134617
[win][arm64ec] Fix msvc-wholearchive for Arm64EC
`msvc-wholearchive` was failing on Arm64EC Windows as it requires the `/MACHINE:ARM64EC` flag to be passed to the MSVC linker.
This required splitting the `extra_c_flags` function into a new `extra_linker_flags` function as `/MACHINE:ARM64EC` is not a valid argument to be passed to the MSVC Compiler (instead, `/arm64EC` should be used).
