`c_variadic`: make `VaList` abi-compatible with C
tracking issue: https://github.com/rust-lang/rust/issues/44930
related PR: rust-lang/rust#144529
On some platforms, the C `va_list` type is actually a single-element array of a struct (on other platforms it is just a pointer). In C, arrays passed as function arguments expirience array-to-pointer decay, which means that C will pass a pointer to the array in the caller instead of the array itself, and modifications to the array in the callee will be visible to the caller (this does not match Rust by-value semantics). However, for `va_list`, the C standard explicitly states that it is undefined behaviour to use a `va_list` after it has been passed by value to a function (in Rust parlance, the `va_list` is moved, not copied). This matches Rust's pass-by-value semantics, meaning that when the C `va_list` type is a single-element array of a struct, the ABI will match C as long as the Rust type is always be passed indirectly.
In the old implementation, this ABI was achieved by having two separate types: `VaList` was the type that needed to be used when passing a `VaList` as a function parameter, whereas `VaListImpl` was the actual `va_list` type that was correct everywhere else. This however is quite confusing, as there are lots of footguns: it is easy to cause bugs by mixing them up (e.g. the C function `void foo(va_list va)` was equivalent to the Rust `fn foo(va: VaList)` whereas the C function `void bar(va_list* va)` was equivalent to the Rust `fn foo(va: *mut VaListImpl)`, not `fn foo(va: *mut VaList)` as might be expected); also converting from `VaListImpl` to `VaList` with `as_va_list()` had platform specific behaviour: on single-element array of a struct platforms it would return a `VaList` referencing the original `VaListImpl`, whereas on other platforms it would return a cioy,
In this PR, there is now just a single `VaList` type (renamed from `VaListImpl`) which represents the C `va_list` type and will just work in all positions. Instead of having a separate type just to make the ABI work, rust-lang/rust#144529 adds a `#[rustc_pass_indirectly_in_non_rustic_abis]` attribute, which when applied to a struct will force the struct to be passed indirectly by non-Rustic calling conventions. This PR then implements the `VaList` rework, making use of the new attribute on all platforms where the C `va_list` type is a single-element array of a struct.
Cleanup of the `VaList` API and implementation is also included in this PR: since it was decided it was OK to experiment with Rust requiring that not calling `va_end` is not undefined behaviour (https://github.com/rust-lang/rust/issues/141524#issuecomment-3028383594), I've removed the `with_copy` method as it was redundant to the `Clone` impl (the `Drop` impl of `VaList` is a no-op as `va_end` is a no-op on all known platforms).
Previous discussion: rust-lang/rust#141524 and [t-compiler > c_variadic API and ABI](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/c_variadic.20API.20and.20ABI)
Tracking issue: https://github.com/rust-lang/rust/issues/44930
r? `@joshtriplett`
misc coercion cleanups and handle safety correctly
r? lcnr
### "remove normalize call"
Fixesrust-lang/rust#132765
If the normalization fails we would sometimes get a `TypeError` containing inference variables created inside of the probe used by coercion. These would then get leaked out causing ICEs in diagnostics logic
### "leak check and lub for closure<->closure coerce-lubs of same defids"
Fixes https://github.com/rust-lang/trait-system-refactor-initiative/issues/233
```rust
fn peculiar() -> impl Fn(u8) -> u8 {
return |x| x + 1
}
```
the `|x| x + 1` expr has a type of `Closure(?31t)` which we wind up inferring the RPIT to. The `CoerceMany` `ret_coercion` for the whole `peculiar` typeck has an expected type of `RPIT` (unnormalized). When we type check the `return |x| x + 1` expr we go from the never type to `Closure(?31t)` which then participates in the `ret_coercion` giving us a `coerce-lub(RPIT, Closure(?31t))`.
Normalizing `RPIT` gives us some `Closure(?50t)` where `?31t` and `?50t` have been unified with `?31t` as the root var. `resolve_vars_if_possible` doesn't resolve infer vars to their roots so these wind up with different structural identities so the fast path doesn't apply and we fall back to coercing to a `fn` ptr. cc rust-lang/rust#147193 which also fixes this
New solver probably just gets more inference variables here because canonicalization + generally different approach to normalization of opaques. Idk :3
### FCP worthy stuffy
there are some other FCP worthy things but they're in my FCP comment which also contains some analysis of the breaking nature of the previously listed changes in this PR: https://github.com/rust-lang/rust/pull/148602#issuecomment-3503497467
Port the `#![windows_subsystem]` attribute to the new attribute system
Part of rust-lang/rust#131229.
I think it's worth running the Windows test suite before merging that (I don't have the rights for this).
Fix MaybeUninit codegen using GVN
This is an alternative to https://github.com/rust-lang/rust/pull/142837, based on https://github.com/rust-lang/rust/pull/146355#discussion_r2421651968.
The general approach I took here is to aggressively propagate anything that is entirely uninitialized. GVN generally takes the approach of only synthesizing small types, but we need to generate large consts to fix the codegen issue.
I also added a special case to MIR dumps for this where now an entirely uninit const is printed as `const <uninit>`, because otherwise we end up with extremely verbose dumps of the new consts.
After GVN though, we still end up with a lot of MIR that looks like this:
```
StorageLive(_1);
_1 = const <uninit>;
_2 = &raw mut _1;
```
Which will break tests/codegen-llvm/maybeuninit-rvo.rs with the naive lowering. I think the ideal fix here is to somehow omit these `_1 = const <uninit>` assignments that come directly after a StorageLive, but I'm not sure how to do that. For now at least, ignoring such assignments (even if they don't come right after a StorageLive) in codegen seems to work.
Note that since GVN is based on synthesizing a `ConstValue` which has a defined layout, this scenario still gets deoptimized by LLVM.
```rust
#![feature(rustc_attrs)]
#![crate_type = "lib"]
use std::mem::MaybeUninit;
#[unsafe(no_mangle)]
pub fn oof() -> [[MaybeUninit<u8>; 8]; 8] {
#[rustc_no_mir_inline]
pub fn inner<T: Copy>() -> [[MaybeUninit<T>; 8]; 8] {
[[MaybeUninit::uninit(); 8]; 8]
}
inner()
}
```
This case can be handled correctly if enough inlining has happened, or it could be handled by post-mono GVN. Synthesizing `UnevaluatedConst` or some other special kind of const seems dubious.
Add `overflow_checks` intrinsic
This adds an intrinsic which allows code in a pre-built library to inherit the overflow checks option from a crate depending on it. This enables code in the standard library to explicitly change behavior based on whether `overflow_checks` are enabled, regardless of the setting used when standard library was compiled.
This is very similar to the `ub_checks` intrinsic, and refactors the two to use a common mechanism.
The primary use case for this is to allow the new `RangeFrom` iterator to yield the maximum element before overflowing, as requested [here](https://github.com/rust-lang/rust/issues/125687#issuecomment-2151118208). This PR includes a working `IterRangeFrom` implementation based on this new intrinsic that exhibits the desired behavior.
[Prior discussion on Zulip](https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/Ability.20to.20select.20code.20based.20on.20.60overflow_checks.60.3F)
Add LLVM realtime sanitizer
This is a new attempt at adding the [LLVM real-time sanitizer](https://clang.llvm.org/docs/RealtimeSanitizer.html) to rust.
Previously this was attempted in https://github.com/rust-lang/rfcs/pull/3766.
Since then the `sanitize` attribute was introduced in https://github.com/rust-lang/rust/pull/142681 and it is a lot more flexible than the old `no_santize` attribute. This allows adding real-time sanitizer without the need for a new attribute, like it was proposed in the RFC. Because i only add a new value to a existing command line flag and to a attribute i don't think an MCP is necessary.
Currently real-time santizer is usable in rust code with the [rtsan-standalone](https://crates.io/crates/rtsan-standalone) crate. This downloads or builds the sanitizer runtime and then links it into the rust binary.
The first commit adds support for more detailed sanitizer information.
The second commit then actually adds real-time sanitizer.
The third adds a warning against using real-time sanitizer with async functions, cloures and blocks because it doesn't behave as expected when used with async functions. I am not sure if this is actually wanted, so i kept it in a seperate commit.
The fourth commit adds the documentation for real-time sanitizer.
Add -Zannotate-moves for profiler visibility of move/copy operations (codegen)
**Note:** this is an alternative implementation of https://github.com/rust-lang/rust/pull/147206; rather than being a MIR transform, it adds the annotations closer to codegen. It's functionally the same but the implementation is lower impact and it could be more correct.
---
This implements a new unstable compiler flag `-Zannotate-moves` that makes move and copy operations visible in profilers by creating synthetic debug information. This is achieved with zero runtime cost by manipulating debug info scopes to make moves/copies appear as calls to `compiler_move<T, SIZE>` and `compiler_copy<T, SIZE>` marker functions in profiling tools.
This allows developers to identify expensive move/copy operations in their code using standard profiling tools, without requiring specialized tooling or runtime instrumentation.
The implementation works at codegen time. When processing MIR operands (`Operand::Move` and `Operand::Copy`), the codegen creates an `OperandRef` with an optional `move_annotation` field containing an `Instance` of the appropriate profiling marker function. When storing the operand, `store_with_annotation()` wraps the store operation in a synthetic debug scope that makes it appear inlined from the marker.
Two marker functions (`compiler_move` and `compiler_copy`) are defined in `library/core/src/profiling.rs`. These are never actually called - they exist solely as debug info anchors.
Operations are only annotated if:
- We're generating debug info and the feature is enabled.
- Meets the size threshold (default: 65 bytes, configurable via `-Zannotate-moves=SIZE`), and is non-zero
- Has a memory representation
This has a very small size impact on object file size. With the default limit it's well under 0.1%, and even with a very small limit of 8 bytes it's still ~1.5%. This could be enabled by default.
Offload device
LLVM's offload functionality usually expects an extra dyn_ptr argument. We could avoid it,b ut likely gonna need it very soon in one of the follow-up PRs (e.g. to request shared memory). So we might as well already add it.
This PR adds a %dyn_ptr ptr to GPUKernel ABI functions, if the offload feature is enabled.
WIP
r? ```@ghost```
This implements a new unstable compiler flag `-Zannotate-moves` that makes
move and copy operations visible in profilers by creating synthetic debug
information. This is achieved with zero runtime cost by manipulating debug
info scopes to make moves/copies appear as calls to `compiler_move<T, SIZE>`
and `compiler_copy<T, SIZE>` marker functions in profiling tools.
This allows developers to identify expensive move/copy operations in their
code using standard profiling tools, without requiring specialized tooling
or runtime instrumentation.
The implementation works at codegen time. When processing MIR operands
(`Operand::Move` and `Operand::Copy`), the codegen creates an `OperandRef`
with an optional `move_annotation` field containing an `Instance` of the
appropriate profiling marker function. When storing the operand,
`store_with_annotation()` wraps the store operation in a synthetic debug
scope that makes it appear inlined from the marker.
Two marker functions (`compiler_move` and `compiler_copy`) are defined
in `library/core/src/profiling.rs`. These are never actually called -
they exist solely as debug info anchors.
Operations are only annotated if the type:
- Meets the size threshold (default: 65 bytes, configurable via
`-Zannotate-moves=SIZE`)
- Has a non-scalar backend representation (scalars use registers,
not memcpy)
This has a very small size impact on object file size. With the default
limit it's well under 0.1%, and even with a very small limit of 8 bytes
it's still ~1.5%. This could be enabled by default.
Add default sanitizers to TargetOptions
Some sanitizers are part of a system's ABI, like the shadow call stack on Aarch64 and RISC-V Fuchsia. Typically ABI options have other spellings, but LLVM has, for historical reasons, marked this as a sanitizer instead of an alternate ABI option. As a result, Fuchsia targets may not be compiled against the correct ABI unless this option is set. This hasn't caused correctness problems, since the backend reserves the SCS register, and thus preserves its value. But this is an issue for unwinding, as the SCS will not be an array of PCs describing the call complete call chain, and will have gaps from callers that don't use the correct ABI.
In the long term, I'd like to see all the sanitizer configs that all frontends copy from clang moved into llvm's libFrontend, and exposed so that frontend consumers can use a small set of simple APIs to use sanitizers in a consistent way across the LLVM ecosystem, but that work is not yet ready today.
Some sanitizers are part of a system's ABI, like the shadow call stack
on Aarch64 and RISC-V Fuchsia. Typically ABI options have other
spellings, but LLVM has, for historical reasons, marked this as a
sanitizer instead of an alternate ABI option. As a result, Fuchsia
targets may not be compiled against the correct ABI unless this option
is set. This hasn't caused correctness problems, since the backend
reserves the SCS register, and thus preserves its value. But this is an
issue for unwinding, as the SCS will not be an array of PCs describing
the call complete call chain, and will have gaps from callers that don't
use the correct ABI.
In the long term, I'd like to see all the sanitizer configs that all
frontends copy from clang moved into llvm's libFrontend, and exposed so
that frontend consumers can use a small set of simple APIs to use
sanitizers in a consistent way across the LLVM ecosystem, but that work
is not yet ready today.
Remove a special case and move another one out of reachable_non_generics
One is no longer necessary, the other is best placed in cg_llvm as it works around a cg_llvm bug.
