naked_asm: emit a label starting with `func_end`
The `cargo asm` tool (`cargo install cargo-show-asm`) pattern matches on such labels to figure out where functions end: normal functions generated by LLVM always do have such a label. We don't guarantee that naked functions emit such a label, but having `cargo asm` work is convenient.
be45f67454/src/asm/statements.rs (L897-L901)
To make the label name unique it's suffixed with the name of the current symbol.
r? ```@Amanieu```
The `cargo asm` tool pattern matches on such labels to figure out where functions end: normal functions generated by LLVM always do have such a label. We don't guarantee that naked functions emit such a label, but having `cargo asm` work is convenient
Clean up `ty::Dynamic`
1. As a follow-up to PR rust-lang/rust#143036, remove `DynKind` entirely.
2. Inside HIR ty lowering, consolidate modules `dyn_compatibility` and `lint` into `dyn_trait`
* `dyn_compatibility` wasn't about dyn compatibility itself, it's about lowering trait object types
* `lint` contained dyn-Trait-specific diagnostics+lints only
Remove Rvalue::Len again.
Now that we have `RawPtrKind::FakeForPtrMetadata`, we can reimplement `Rvalue::Len` using `PtrMetadata(&raw const (fake) place)`.
r? ``@scottmcm``
interpret: fix overlapping aggregate initialization
This fixes the problem pointed out by ````@saethlin```` in https://github.com/rust-lang/rust/issues/146383#issuecomment-3273224645.
Also clarify when exactly current de-facto MIR semantics allow overlap of the LHS and RHS in an assignment.
match clang's `va_arg` assembly on arm targets
tracking issue: https://github.com/rust-lang/rust/issues/44930
For this example
```rust
#![feature(c_variadic)]
#[unsafe(no_mangle)]
unsafe extern "C" fn variadic(a: f64, mut args: ...) -> f64 {
let b = args.arg::<f64>();
let c = args.arg::<f64>();
a + b + c
}
```
We currently generate (via llvm):
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d0, r0, r1
add r0, sp, #4
vldr d1, [sp, #4]
add r0, r0, #15
bic r0, r0, #7
vadd.f64 d0, d0, d1
add r1, r0, #8
str r1, [sp]
vldr d1, [r0]
vadd.f64 d0, d0, d1
vmov r0, r1, d0
add sp, sp, #12
bx lr
```
LLVM is not doing a good job. In fact, it's well-known that LLVM's implementation of `va_arg` is kind of bad, and we implement it ourselves (based on clang) for many targets already. For arm, our own `emit_ptr_va_arg` saves 3 instructions.
Next, it turns out it's important for LLVM to explicitly start and end the lifetime of the `va_list`. In https://github.com/rust-lang/rust/pull/146059 I already end the lifetime, but when looking at this again, I noticed that it is important to also start it, see https://godbolt.org/z/EGqvKTTsK: failing to explicitly start the lifetime uses an extra register.
So, the combination of `emit_ptr_va_arg` with starting/ending the lifetime makes rustc emit exactly the instructions that clang generates::
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d16, r0, r1
vldr d17, [sp, #4]
vadd.f64 d16, d16, d17
vldr d17, [sp, #12]
vadd.f64 d16, d16, d17
vmov r0, r1, d16
add sp, sp, #12
bx lr
```
The arguments to `emit_ptr_va_arg` are based on [the clang implementation](03dc2a41f3/clang/lib/CodeGen/Targets/ARM.cpp (L798-L844)).
r? ``@workingjubilee`` (I can re-roll if your queue is too full, but you do seem like the right person here)
try-job: armhf-gnu
Fix tail calls to `#[track_caller]` functions
We want `#[track_caller]` to be semver independent, i.e. it should not be a breaking change to add or remove it. Since it changes ABI of a function (adding an additional argument) we have to be careful to preserve this property when adding tail calls.
The only way to achieve this that I can see is:
- we forbid tail calls in functions which are marked with `#[track_caller]` (already implemented)
- tail-calling a `#[track_caller]` marked function downgrades the tail-call to a normal call (or equivalently tail-calls the shim made by fn def to fn ptr cast) (this pr)
Ideally the downgrade would be performed by a MIR pass, but that requires post mono MIR opts (cc ```@saethlin,``` rust-lang/rust#131650). For now I've changed code in cg_ssa to accomodate this behaviour (+ added a hack to mono collector so that the shim is actually generated)
Additionally I added a lint, although I don't think it's strictly necessary.
Alternative to rust-lang/rust#144762 (and thus closesrust-lang/rust#144762)
Fixes https://github.com/rust-lang/rust/issues/144755
atomicrmw on pointers: move integer-pointer cast hacks into backend
Conceptually, we want to have atomic operations on pointers of the form `fn atomic_add(ptr: *mut T, offset: usize, ...)`. However, LLVM does not directly support such operations (https://github.com/llvm/llvm-project/issues/120837), so we have to cast the `offset` to a pointer somewhere.
This PR moves that hack into the LLVM backend, so that the standard library, intrinsic, and Miri all work with the conceptual operation we actually want. Hopefully, one day LLVM will gain a way to represent these operations without integer-pointer casts, and then the hack will disappear entirely.
Cc ```@nikic``` -- this is the best we can do right now, right?
Fixes https://github.com/rust-lang/rust/issues/134617
coverage: Remove all unstable support for MC/DC instrumentation
Preliminary support for a partial implementation of “Modified Condition/Decision Coverage” instrumentation was added behind the unstable flag `-Zcoverage-options=mcdc` in 2024. These are the most substantial PRs involved:
- rust-lang/rust#123409
- rust-lang/rust#126733
At the time, I accepted these PRs with relatively modest scrutiny, because I did not want to stand in the way of independent work on MC/DC instrumentation. My hope was that ongoing work by interested contributors would lead to the code becoming clearer and more maintainable over time.
---
However, that MC/DC code has proven itself to be a major burden on overall maintenance of coverage instrumentation, and a major obstacle to other planned improvements, such as internal changes needed for proper support of macro expansion regions.
I have also become reluctant to accept any further MC/DC-related changes that would increase this burden.
That tension has resulted in an unhappy impasse. On one hand, the present MC/DC implementation is not yet complete, and shows little sign of being complete at an acceptable level of code quality in the foreseeable future. On the other hand, the continued existence of this partial MC/DC implementation is imposing serious maintenance burdens on every other aspect of coverage instrumentation, and is preventing some of the very improvements that would make it easier to accept expanded MC/DC support in the future.
While I know this will be disappointing to some, I think the healthy way forward is accept that I made the wrong call in accepting the current implementation, and to remove it entirely from the compiler.
Implement support for `become` and explicit tail call codegen for the LLVM backend
This PR implements codegen of explicit tail calls via `become` in `rustc_codegen_ssa` and support within the LLVM backend. Completes a task on (https://github.com/rust-lang/rust/issues/112788). This PR implements all the necessary bits to make explicit tail calls usable, other backends have received stubs for now and will ICE if you use `become` on them. I suspect there is some bikeshedding to be done on how we should go about implementing this for other backends, but it should be relatively straightforward for GCC after this is merged.
During development I also put together a POC bytecode VM based on tail call dispatch to test these changes out and analyze the codegen to make sure it generates expected assembly. That is available [here](https://github.com/xacrimon/tcvm).
No longer need `alloca`s for consuming `Result<!, i32>` and similar
In optimized builds GVN gets rid of these already, but in `opt-level=0` we actually make `alloca`s for this, which particularly impacts `?`-style things that use actually-only-one-variant types like this.
While doing so, rewrite `LocalAnalyzer::process_place` to be non-recursive, solving a 6+ year old FIXME.
r? codegen
Let `codegen_transmute_operand` just handle everything
When combined with rust-lang/rust#143720, this means `rvalue_creates_operand` can just return `true` for *every* `Rvalue`. (A future PR could consider removing it, though just letting it optimize out is fine for now.)
It's nicer anyway, IMHO, because it avoids needing the layout checks to be consistent in the two places, and thus is an overall reduction in code. Plus it's a more helpful building block when used in other places this way.
(TBH, it probably would have been better to have it this way the whole time, but I clearly didn't understand `rvalue_creates_operand` when I originally wrote rust-lang/rust#109843.)
Don't emit two `assume`s in transmutes when one is a subset of the other
For example, transmuting between `bool` and `Ordering` doesn't need two `assume`s because one range is a superset of the other.
Multiple are still used for things like `char` <-> `NonZero<u32>`, which overlap but where neither fully contains the other.
For example, transmuting between `bool` and `Ordering` doesn't need two `assume`s because one range is a superset of the other.
Multiple are still used for things like `char` <-> `NonZero<u32>`, which overlap but where neither fully contains the other.
When combined with 143720, this means `rvalue_creates_operand` can just return `true` for *every* `Rvalue`. (A future PR could consider removing it, though just letting it optimize out is fine for now.)
It's nicer anyway, IMHO, because it avoids needing the layout checks to be consistent in the two places, and thus is an overall reduction in code. Plus it's a more helpful building block when used in other places this way.
In optimized builds GVN gets rid of these already, but in `opt-level=0` we actually make `alloca`s for this, which particularly impacts `?`-style things that use actually-only-one-variant types like this.
The conversation in 143502 made be realize how easy this is to handle, since the only possibilty is ZSTs -- everything else ends up with the destination being `LocalKind::Memory` and thus doesn't call `codegen_rvalue_operand` at all.
This gets us perilously close to a world where `rvalue_creates_operand` only ever returns true. I'll try out such a world next :)
Simplify discriminant codegen for niche-encoded variants which don't wrap across an integer boundary
Inspired by rust-lang/rust#139729, this attempts to be a much-simpler and more-localized change while still making a difference. (Specifically, this does not try to solve the problem with select-sinking, leaving that to be fixed by https://github.com/llvm/llvm-project/issues/134024 -- once it gets released -- instead of in rustc's codegen.)
What this *does* improve is checking for the variant in a 3+ variant enum when that variant is the type providing the niche. Something like `if let Foo::WithBool(_) = ...` previously compiled to `ugt(add(x, -2), 2)`, which is non-trivial to think about because it's depending on the unsigned wrapping to shift the 0/1 up above 2. With this PR it compiles to just `ult(x, 2)`, which is probably what you'd have written yourself if you were doing it by hand to look for "is this byte a bool?".
That's done by leaving most of the codegen alone, but adding a couple new special cases to the `is_niche` check. The default looks at the relative discriminant, but in the common cases where there's no wraparound involved, we can just check the original value, rather than the offsetted one.
The first commit just adds some tests, so the best way to see the effect of this change is to look at the second commit and how it updates the test expectations.
Make UB transmutes really UB in LLVM
Ralf suggested in <https://github.com/rust-lang/rust/pull/143410#discussion_r2184928123> that UB transmutes shouldn't be trapping, which happened for the one path *that* PR was changing, but there's another path as well, so *this* PR changes that other path to match.
r? codegen
Let `rvalue_creates_operand` return true for *all* `Rvalue::Aggregate`s
~~Draft for now because it's built on Ralf's rust-lang/rust#143291~~
Inspired by https://github.com/rust-lang/rust/pull/138759#discussion_r2156375342 where I noticed that we were nearly at this point, plus the comments I was writing in rust-lang/rust#143410 that reminded me a type-dependent `true` is fine.
This PR splits the `OperandRef::builder` logic out to a separate type, with the updates needed to handle SIMD as well. In doing so, that makes the existing `Aggregate` path in `codegen_rvalue_operand` capable of handing SIMD values just fine.
As a result, we no longer need to do layout calculations for aggregate result types when running the analysis to determine which things can be SSA in codegen.
