`codegen_switchint_terminator` already uses `br` instead of `switch`
when there is one normal target plus the `otherwise` target. But there's
another common case with two normal targets and an `otherwise` target
that points to an empty unreachable BB. This comes up a lot when
switching on the tags of enums that use niches.
The pattern looks like this:
```
bb1: ; preds = %bb6
%3 = load i8, ptr %_2, align 1, !range !9, !noundef !4
%4 = sub i8 %3, 2
%5 = icmp eq i8 %4, 0
%_6 = select i1 %5, i64 0, i64 1
switch i64 %_6, label %bb3 [
i64 0, label %bb4
i64 1, label %bb2
]
bb3: ; preds = %bb1
unreachable
```
This commit adds code to convert the `switch` to a `br`:
```
bb1: ; preds = %bb6
%3 = load i8, ptr %_2, align 1, !range !9, !noundef !4
%4 = sub i8 %3, 2
%5 = icmp eq i8 %4, 0
%_6 = select i1 %5, i64 0, i64 1
%6 = icmp eq i64 %_6, 0
br i1 %6, label %bb4, label %bb2
bb3: ; No predecessors!
unreachable
```
This has a surprisingly large effect on compile times, with reductions
of 5% on debug builds of some crates. The reduction is all due to LLVM
taking less time. Maybe LLVM is just much better at handling `br` than
`switch`.
The resulting code is still suboptimal.
- The `icmp`, `select`, `icmp` sequence is silly, converting an `i1` to an `i64`
and back to an `i1`. But with the current code structure it's hard to avoid,
and LLVM will easily clean it up, in opt builds at least.
- `bb3` is usually now truly dead code (though not always, so it can't
be removed universally).
Don't use usub.with.overflow intrinsic
The canonical form of a usub.with.overflow check in LLVM are separate sub + icmp instructions, rather than a usub.with.overflow intrinsic. Using usub.with.overflow will generally result in worse optimization potential.
The backend will attempt to form usub.with.overflow when it comes to actual instruction selection. This is not fully reliable, but I believe this is a better tradeoff than using the intrinsic in IR.
Fixes#103285.
Fix line numbers for MIR inlined code
`should_collapse_debuginfo` detects if the specified span is part of a
macro expansion however it does this by checking if the span is anything
other than a normal (non-expanded) kind, then the span sequence is
walked backwards to the root span.
This doesn't work when the MIR inliner inlines code as it creates spans
with expansion information set to `ExprKind::Inlined` and results in the
line number being attributed to the inline callsite rather than the
normal line number of the inlined code.
Fixes#103068
indirect immutable freeze by-value function parameters.
Right now, `rustc` only examines function signatures and the platform ABI when
determining the LLVM attributes to apply to parameters. This results in missed
optimizations, because there are some attributes that can be determined via
analysis of the MIR making up the function body. In particular, `readonly`
could be applied to most indirectly-passed by-value function arguments
(specifically, those that are freeze and are observed not to be mutated), but
it currently is not.
This patch introduces the machinery that allows `rustc` to determine those
attributes. It consists of a query, `deduced_param_attrs`, that, when
evaluated, analyzes the MIR of the function to determine supplementary
attributes. The results of this query for each function are written into the
crate metadata so that the deduced parameter attributes can be applied to
cross-crate functions. In this patch, we simply check the parameter for
mutations to determine whether the `readonly` attribute should be applied to
parameters that are indirect immutable freeze by-value. More attributes could
conceivably be deduced in the future: `nocapture` and `noalias` come to mind.
Adding `readonly` to indirect function parameters where applicable enables some
potential optimizations in LLVM that are discussed in [issue 103103] and [PR
103070] around avoiding stack-to-stack memory copies that appear in functions
like `core::fmt::Write::write_fmt` and `core::panicking::assert_failed`. These
functions pass a large structure unchanged by value to a subfunction that also
doesn't mutate it. Since the structure in this case is passed as an indirect
parameter, it's a pointer from LLVM's perspective. As a result, the
intermediate copy of the structure that our codegen emits could be optimized
away by LLVM's MemCpyOptimizer if it knew that the pointer is `readonly
nocapture noalias` in both the caller and callee. We already pass `nocapture
noalias`, but we're missing `readonly`, as we can't determine whether a
by-value parameter is mutated by examining the signature in Rust. I didn't have
much success with having LLVM infer the `readonly` attribute, even with fat
LTO; it seems that deducing it at the MIR level is necessary.
No large benefits should be expected from this optimization *now*; LLVM needs
some changes (discussed in [PR 103070]) to more aggressively use the `noalias
nocapture readonly` combination in its alias analysis. I have some LLVM patches
for these optimizations and have had them looked over. With all the patches
applied locally, I enabled LLVM to remove all the `memcpy`s from the following
code:
```rust
fn main() {
println!("Hello {}", 3);
}
```
which is a significant codegen improvement over the status quo. I expect that
if this optimization kicks in in multiple places even for such a simple
program, then it will apply to Rust code all over the place.
[issue 103103]: https://github.com/rust-lang/rust/issues/103103
[PR 103070]: https://github.com/rust-lang/rust/pull/103070
The canonical form of a usub.with.overflow check in LLVM are
separate sub + icmp instructions, rather than a usub.with.overflow
intrinsic. Using usub.with.overflow will generally result in worse
optimization potential.
The backend will attempt to form usub.with.overflow when it comes
to actual instruction selection. This is not fully reliable, but
I believe this is a better tradeoff than using the intrinsic in
IR.
Fixes#103285.
`should_collapse_debuginfo` detects if the specified span is part of a
macro expansion however it does this by checking if the span is anything
other than a normal (non-expanded) kind, then the span sequence is
walked backwards to the root span.
This doesn't work when the MIR inliner inlines code as it creates spans
with expansion information set to `ExprKind::Inlined` and results in the
line number being attributed to the inline callsite rather than the
normal line number of the inlined code.
Fix the sanitizer_scs_attr_check.rs test
The test is failing when targeting aarch64 Android. The intent appears to have been to look for a function attributes comment (or the absence of one) on the line preceding the function declaration. But this isn't quite possible with FileCheck and the test as written was looking for a line with `no_scs` after a line with `scs`, which doesn't appear in the output. Instead, match on the function attributes comment on the line following the demangled function name comment.
The test is failing when targeting aarch64 Android. The intent appears
to have been to look for a function attributes comment (or the absence
of one) on the line preceding the function declaration. But this isn't
quite possible with FileCheck and the test as written was looking for a
line with `no_scs` after a line with `scs`, which doesn't appear in the
output. Instead, match on the function attributes comment on the line
following the demangled function name comment.
Enable inline stack probes on X86 with LLVM 16
The known problems with x86 inline-asm stack probes have been solved on LLVM main (16), so this flips the switch. Anyone using bleeding-edge LLVM with rustc can start testing this, as I have done locally. We'll get more direct rust-ci when LLVM 16 branches and we start our upgrade, and we can always patch or disable it then if we find new problems.
The previous attempt was #77885, reverted in #84708.
Declare `main` as visibility hidden on targets that default to hidden.
On targets with `default_hidden_visibility` set, which is currrently just WebAssembly, declare the generated `main` function with visibility hidden. This makes it consistent with clang's WebAssembly target, where `main` is just a user function that gets the same visibility as any other user function, which is hidden on WebAssembly unless explicitly overridden.
This will help simplify use cases which in the future may want to automatically wasm-export all visibility-"default" symbols. `main` isn't intended to be wasm-exported, and marking it hidden prevents it from being wasm-exported in that scenario.
Optimize `array::IntoIter`
`.into_iter()` on arrays was slower than it needed to be (especially compared to slice iterator) since it uses `Range<usize>`, which needs to handle degenerate ranges like `10..4`.
This PR adds an internal `IndexRange` type that's like `Range<usize>` but with a safety invariant that means it doesn't need to worry about those cases -- it only handles `start <= end` -- and thus can give LLVM more information to optimize better.
I added one simple demonstration of the improvement as a codegen test.
(`vec::IntoIter` uses pointers instead of indexes, so doesn't have this problem, but that only works because its elements are boxed. `array::IntoIter` can't use pointers because that would keep it from being movable.)
`.into_iter()` on arrays was slower than it needed to be (especially compared to slice iterator) since it uses `Range<usize>`, which needs to handle degenerate ranges like `10..4`.
This PR adds an internal `IndexRange` type that's like `Range<usize>` but with a safety invariant that means it doesn't need to worry about those cases -- it only handles `start <= end` -- and thus can give LLVM more information to optimize better.
I added one simple demonstration of the improvement as a codegen test.
Fix a bunch of typo
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
Add pointer masking convenience functions
This PR adds the following public API:
```rust
impl<T: ?Sized> *const T {
fn mask(self, mask: usize) -> *const T;
}
impl<T: ?Sized> *mut T {
fn mask(self, mask: usize) -> *const T;
}
// mod intrinsics
fn mask<T>(ptr: *const T, mask: usize) -> *const T
```
This is equivalent to `ptr.map_addr(|a| a & mask)` but also uses a cool llvm intrinsic.
Proposed in https://github.com/rust-lang/rust/pull/95643#issuecomment-1121562352
cc `@Gankra` `@scottmcm` `@RalfJung`
r? rust-lang/libs-api
Make forward compatibility lint deprecated_cfg_attr_crate_type_name deny by default
Turns the forward compatibility lint added by #83744 to deprecate `cfg_attr` usage with `#![crate_type]` and `#![crate_name]` attributes into deny by default. Copying the example from #83744:
```Rust
#![crate_type = "lib"] // remains working
#![cfg_attr(foo, crate_type = "bin")] // will stop working
```
Over 8 months have passed since #83744 was merged so I'd say this gives ample time for people to have been warned, so we can make the warning stronger. No usage was found via grep.app except for one, which was in an unmaintained code base that didn't seem to be used in the open source eco system. The crater run conducted in #83744 also didn't show up anything.
cc #91632 - tracking issue for the lint
I couldn't find where exactly it's documented, but apperantly pointers to void
type are invalid in llvm - void is only allowed as a return type of functions.
Add LLVM15-specific codegen test for `try`/`?`s that now optimize away
These still generated a bunch of code back in Rust 1.63 (<https://rust.godbolt.org/z/z31P8h6rz>), but with LLVM 15 merged they no longer do 🎉
