Monomorphise `#[repr(transparent)]` parameterized ADTs before turning
them into an Itanium mangled String.
`#[repr(transparent)]` ADTs currently use the single field to represent
them in their CFI type ID to ensure that they are compatible. However,
if that type involves a type parameter instantiated at the ADT level, as
in `ManuallyDrop`, this will currently ICE as the `Parameter` type
cannot be mangled. Since this happens at lowering time, it should always
be concrete after substitution.
Fixes#106230
abort immediately on bad mem::zeroed/uninit
Now that we have non-unwinding panics, let's use them for these assertions. This re-establishes the property that `mem::uninitialized` and `mem::zeroed` will never unwind -- the earlier approach of causing panics here sometimes led to hard-to-debug segfaults when the surrounding code was not able to cope with the unexpected unwinding.
Cc `@bjorn3` I did not touch cranelift but I assume it needs a similar patch. However it has a `codegen_panic` abstraction that I did not want to touch since I didn't know how else it is used.
Codegen test for derived `<` on trivial newtype [TEST ONLY]
I originally wrote this for #106065, but the libcore changes there aren't necessarily a win.
So I pulled out this test to be its own PR since it's important (see https://github.com/rust-lang/rust/pull/105840#discussion_r1056030324) and well-intentioned changes to core or the derive could accidentally break it without that being obvious (other than by massive unexplained perf changes).
https://reviews.llvm.org/D140192 caused the LLVM variable generated
for enum discriminant checks to be named differently (%narrow vs %1).
This adjusts the test CHECK directives to match any name.
Don't internalize __llvm_profile_counter_bias
Currently, LLVM profiling runtime counter relocation cannot be used by rust during LTO because symbols are being internalized before all symbol information is known.
This mode makes LLVM emit a __llvm_profile_counter_bias symbol which is referenced by the profiling initialization, which itself is pulled in by the rust driver here [1].
It is enabled with -Cllvm-args=-runtime-counter-relocation for platforms which are opt-in to this mode like Linux. On these platforms there will be no link error, rather just surprising behavior for a user which request runtime counter relocation. The profiling runtime will not see that symbol go on as if it were never there. On Fuchsia, the profiling runtime must have this symbol which will cause a hard link error.
As an aside, I don't have enough context as to why rust's LTO model is how it is. AFAICT, the internalize pass is only safe to run at link time when all symbol information is actually known, this being an example as to why. I think special casing this symbol as a known one that LLVM can emit which should not have it's visbility de-escalated should be fine given how seldom this pattern of defining an undefined symbol to get initilization code pulled in is. From a quick grep, __llvm_profile_runtime is the only symbol that rustc does this for.
[1] 0265a3e93b/compiler/rustc_codegen_ssa/src/back/linker.rs (L598)
Fix invalid codegen during debuginfo lowering
In order for LLVM to correctly generate debuginfo for msvc, we sometimes need to spill arguments to the stack and perform some direct & indirect offsets into the value. Previously, this code always performed those actions, even when not required as LLVM would clean it up during optimization.
However, when MIR inlining is enabled, this can cause problems as the operations occur prior to the spilled value being initialized. To solve this, we first calculate the necessary offsets using just the type which is side-effect free and does not alter the LLVM IR. Then, if we are in a situation which requires us to generate the LLVM IR (and this situation only occurs for arguments, not local variables) then we perform the same calculation again, this time generating the appropriate LLVM IR as we go.
r? `@tmiasko` but feel free to reassign if you want 🙂Fixes#105386
Fix failing codegen tests on s390x
Several codegen tests are currently failing due to making assumptions that are not valid for the s390x architecture:
- catch-unwind.rs: fails due to inlining differences. Already ignored on another platform for the same reason. Solution: Ignore on s390x.
- remap_path_prefix/main.rs: fails due to different alignment requirement for string constants. Solution: Do not test for the alignment requirement.
- repr-transparent-aggregates-1.rs: many ABI assumptions. Already ignored on many platforms for the same reason. Solution: Ignore on s390x.
- repr-transparent.rs: no vector ABI by default on s390x. Already ignored on another platform for a similar reason. Solution: Ignore on s390x.
- uninit-consts.rs: hard-coded little-endian constant. Solution: Match both little- and big-endian versions.
Fixes part of https://github.com/rust-lang/rust/issues/105383.
Add LLVM KCFI support to the Rust compiler
This PR adds LLVM Kernel Control Flow Integrity (KCFI) support to the Rust compiler. It initially provides forward-edge control flow protection for operating systems kernels for Rust-compiled code only by aggregating function pointers in groups identified by their return and parameter types. (See llvm/llvm-project@cff5bef.)
Forward-edge control flow protection for C or C++ and Rust -compiled code "mixed binaries" (i.e., for when C or C++ and Rust -compiled code share the same virtual address space) will be provided in later work as part of this project by identifying C char and integer type uses at the time types are encoded (see Type metadata in the design document in the tracking issue #89653).
LLVM KCFI can be enabled with -Zsanitizer=kcfi.
Thank you again, `@bjorn3,` `@eddyb,` `@nagisa,` and `@ojeda,` for all the help!
In order for LLVM to correctly generate debuginfo for msvc, we sometimes
need to spill arguments to the stack and perform some direct & indirect
offsets into the value. Previously, this code always performed those
actions, even when not required as LLVM would clean it up during
optimization.
However, when MIR inlining is enabled, this can cause problems as the
operations occur prior to the spilled value being initialized. To solve
this, we first calculate the necessary offsets using just the type which
is side-effect free and does not alter the LLVM IR. Then, if we are in a
situation which requires us to generate the LLVM IR (and this situation
only occurs for arguments, not local variables) then we perform the same
calculation again, this time generating the appropriate LLVM IR as we
go.
This commit adds LLVM Kernel Control Flow Integrity (KCFI) support to
the Rust compiler. It initially provides forward-edge control flow
protection for operating systems kernels for Rust-compiled code only by
aggregating function pointers in groups identified by their return and
parameter types. (See llvm/llvm-project@cff5bef.)
Forward-edge control flow protection for C or C++ and Rust -compiled
code "mixed binaries" (i.e., for when C or C++ and Rust -compiled code
share the same virtual address space) will be provided in later work as
part of this project by identifying C char and integer type uses at the
time types are encoded (see Type metadata in the design document in the
tracking issue #89653).
LLVM KCFI can be enabled with -Zsanitizer=kcfi.
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
Currently, LLVM profiling runtime counter relocation cannot be
used by rust during LTO because symbols are being internalized
before all symbol information is known.
This mode makes LLVM emit a __llvm_profile_counter_bias symbol
which is referenced by the profiling initialization, which itself
is pulled in by the rust driver here [1].
It is enabled with -Cllvm-args=-runtime-counter-relocation for
platforms which are opt-in to this mode like Linux. On these
platforms there will be no link error, rather just surprising
behavior for a user which request runtime counter relocation.
The profiling runtime will not see that symbol go on as if it
were never there. On Fuchsia, the profiling runtime must have
this symbol which will cause a hard link error.
As an aside, I don't have enough context as to why rust's LTO
model is how it is. AFAICT, the internalize pass is only safe
to run at link time when all symbol information is actually
known, this being an example as to why. I think special casing
this symbol as a known one that LLVM can emit which should not
have it's visbility de-escalated should be fine given how
seldom this pattern of defining an undefined symbol to get
initilization code pulled in is. From a quick grep,
__llvm_profile_runtime is the only symbol that rustc does this
for.
[1] 0265a3e93b/compiler/rustc_codegen_ssa/src/back/linker.rs (L598)
Several codegen tests are currently failing due to making
assumptions that are not valid for the s390x architecture:
- catch-unwind.rs: fails due to inlining differences.
Already ignored on another platform for the same reason.
Solution: Ignore on s390x.
- remap_path_prefix/main.rs: fails due to different alignment
requirement for string constants.
Solution: Do not test for the alignment requirement.
- repr-transparent-aggregates-1.rs: many ABI assumptions.
Already ignored on many platforms for the same reason.
Solution: Ignore on s390x.
- repr-transparent.rs: no vector ABI by default on s390x.
Already ignored on another platform for a similar reason.
Solution: Ignore on s390x.
- uninit-consts.rs: hard-coded little-endian constant.
Solution: Match both little- and big-endian versions.
Fixes part of https://github.com/rust-lang/rust/issues/105383.
rustc_codegen_ssa: Fix for codegen_get_discr
When doing the optimized implementation of getting the discriminant, the arithmetic needs to be done in the tag type so wrapping behavior works correctly.
Fixes#104519
Add a codegen-test that verifies inter-crate linkage with the static
relocation model. We expect all symbols that are part of a rust
compilation to end up in the same DSO, thus we expect `dso_local`
annotations.
Stop peeling the last iteration of the loop in `Vec::resize_with`
`resize_with` uses the `ExtendWith` code that peels the last iteration:
341d8b8a2c/library/alloc/src/vec/mod.rs (L2525-L2529)
But that's kinda weird for `ExtendFunc` because it does the same thing on the last iteration anyway:
341d8b8a2c/library/alloc/src/vec/mod.rs (L2494-L2502)
So this just has it use the normal `extend`-from-`TrustedLen` code instead.
r? `@ghost`
Manually implement PartialEq for Option<T> and specialize non-nullable types
This PR manually implements `PartialEq` and `StructuralPartialEq` for `Option`, which seems to produce slightly better codegen than the automatically derived implementation.
It also allows specializing on the `core::num::NonZero*` and `core::ptr::NonNull` types, taking advantage of the niche optimization by transmuting the `Option<T>` to `T` to be compared directly, which can be done in just two instructions.
A comparison of the original, new and specialized code generation is available [here](https://godbolt.org/z/dE4jxdYsa).
Previously, async constructs would be lowered to "normal" generators,
with an additional `from_generator` / `GenFuture` shim in between to
convert from `Generator` to `Future`.
The compiler will now special-case these generators internally so that
async constructs will *directly* implement `Future` without the need
to go through the `from_generator` / `GenFuture` shim.
The primary motivation for this change was hiding this implementation
detail in stack traces and debuginfo, but it can in theory also help
the optimizer as there is less abstractions to see through.
`VecDeque::resize` should re-use the buffer in the passed-in element
Today it always copies it for *every* appended element, but one of those clones is avoidable.
This adds `iter::repeat_n` (https://github.com/rust-lang/rust/issues/104434) as the primitive needed to do this. If this PR is acceptable, I'll also use this in `Vec` rather than its custom `ExtendElement` type & infrastructure that is harder to share between multiple different containers:
101e1822c3/library/alloc/src/vec/mod.rs (L2479-L2492)
When doing the optimized implementation of getting the discriminant, the
arithmetic needs to be done in the tag type so wrapping behavior works
correctly.
Fixes#104519
Fix Access Violation when using lld & ThinLTO on windows-msvc
Users report an AV at runtime of the compiled binary when using lld and ThinLTO on windows-msvc. The AV occurs when accessing a static value which is defined in one crate but used in another. Based on the disassembly of the cross-crate use, it appears that the use is not correctly linked with the definition and is instead assigned a garbage pointer value.
If we look at the symbol tables for each crates' obj file, we can see what is happening:
*lib.obj*:
```
COFF SYMBOL TABLE
...
00E 00000000 SECT2 notype External | _ZN10reproducer7memrchr2FN17h612b61ca0e168901E
...
```
*bin.obj*:
```
COFF SYMBOL TABLE
...
010 00000000 UNDEF notype External | __imp__ZN10reproducer7memrchr2FN17h612b61ca0e168901E
...
```
The use of the symbol has the "import" style symbol name but the declaration doesn't generate any symbol with the same name. As a result, linking the files generates a warning from lld:
> rust-lld: warning: bin.obj: locally defined symbol imported: reproducer::memrchr::FN::h612b61ca0e168901 (defined in lib.obj) [LNK4217]
and the symbol reference remains undefined at runtime leading to the AV.
To fix this, we just need to detect that we are performing ThinLTO (and thus, static linking) and omit the `dllimport` attribute on the extern item in LLVM IR.
Fixes#81408
