De-abuse TyKind::Error in exhaustiveness checking
Replaces https://github.com/rust-lang/rust/pull/71074. Context: https://github.com/rust-lang/rust/issues/70866.
In order to remove the use of `TyKind::Error`, I had to make sure we skip over those fields whose inhabitedness should not be observed. This is potentially error-prone however, since we must be careful not to mix filtered and unfiltered lists of patterns. I managed to hide away most of the filtering behind a new `Fields` struct, that I used everywhere relevant. I quite like the result; I think the twin concepts of `Constructor` and `Fields` make a good mental model.
As usual, I tried to separate commits that shuffle code around from commits that require more thought to review.
cc @varkor @Centril
Experimentally add `ffi_const` and `ffi_pure` extern fn attributes
Add FFI function attributes corresponding to clang/gcc/... `const` and `pure`.
Rebased version of #58327 by @gnzlbg with the following changes:
- Switched back from the `c_ffi_const` and `c_ffi_pure` naming to `ffi_const` and `ffi_pure`, as I agree with https://github.com/rust-lang/rust/pull/58327#issuecomment-462718772 and this nicely aligns with `ffi_returns_twice`
- (Hopefully) took care of all of @hanna-kruppe's change requests in the original PR
r? @hanna-kruppe
Dumb NRVO
This is a very simple version of an NRVO pass, which scans backwards from the `return` terminator to see if there is an an assignment like `_0 = _1`. If a basic block with two or more predecessors is encountered during this scan without first seeing an assignment to the return place, we bail out. This avoids running a full "reaching definitions" dataflow analysis.
I wanted to see how much `rustc` would benefit from even a very limited version of this optimization. We should be able to use this as a point of comparison for more advanced versions that are based on live ranges.
r? @ghost
Use `T`'s discriminant type in `mem::Discriminant<T>` instead of `u64`.
fixes#70509
Adds the lang-item `discriminant_kind`.
Updates the function signature of `intrinsics::discriminant_value`.
Adds the *probably permanently unstable* trait `DiscriminantKind`.
`mem::Discriminant` should now be smaller in some cases.
r? @ghost
Make `fold` standalone.
`fold` is currently implemented via `try_fold`, but implementing it
directly results in slightly less LLVM IR being generated, speeding up
compilation of some benchmarks.
r? @cuviper
Suggest fixes and add error recovery for `use foo::self`
Fixes#63741.
I have implemented 2 suggestions on how to fix a `use foo::self` import, however I feel like showing them both might be too verbose.
Additionally, I have also implemented error recovery as [menitoned](https://github.com/rust-lang/rust/issues/63741#issuecomment-602391091) by @comex.
I believe r? @estebank deals with diagnostics.
Implement new asm! syntax from RFC 2850
This PR implements the new `asm!` syntax proposed in https://github.com/rust-lang/rfcs/pull/2850.
# Design
A large part of this PR revolves around taking an `asm!` macro invocation and plumbing it through all of the compiler layers down to LLVM codegen. Throughout the various stages, an `InlineAsm` generally consists of 3 components:
- The template string, which is stored as an array of `InlineAsmTemplatePiece`. Each piece represents either a literal or a placeholder for an operand (just like format strings).
```rust
pub enum InlineAsmTemplatePiece {
String(String),
Placeholder { operand_idx: usize, modifier: Option<char>, span: Span },
}
```
- The list of operands to the `asm!` (`in`, `[late]out`, `in[late]out`, `sym`, `const`). These are represented differently at each stage of lowering, but follow a common pattern:
- `in`, `out` and `inout` all have an associated register class (`reg`) or explicit register (`"eax"`).
- `inout` has 2 forms: one with a single expression that is both read from and written to, and one with two separate expressions for the input and output parts.
- `out` and `inout` have a `late` flag (`lateout` / `inlateout`) to indicate that the register allocator is allowed to reuse an input register for this output.
- `out` and the split variant of `inout` allow `_` to be specified for an output, which means that the output is discarded. This is used to allocate scratch registers for assembly code.
- `sym` is a bit special since it only accepts a path expression, which must point to a `static` or a `fn`.
- The options set at the end of the `asm!` macro. The only one that is particularly of interest to rustc is `NORETURN` which makes `asm!` return `!` instead of `()`.
```rust
bitflags::bitflags! {
pub struct InlineAsmOptions: u8 {
const PURE = 1 << 0;
const NOMEM = 1 << 1;
const READONLY = 1 << 2;
const PRESERVES_FLAGS = 1 << 3;
const NORETURN = 1 << 4;
const NOSTACK = 1 << 5;
}
}
```
## AST
`InlineAsm` is represented as an expression in the AST:
```rust
pub struct InlineAsm {
pub template: Vec<InlineAsmTemplatePiece>,
pub operands: Vec<(InlineAsmOperand, Span)>,
pub options: InlineAsmOptions,
}
pub enum InlineAsmRegOrRegClass {
Reg(Symbol),
RegClass(Symbol),
}
pub enum InlineAsmOperand {
In {
reg: InlineAsmRegOrRegClass,
expr: P<Expr>,
},
Out {
reg: InlineAsmRegOrRegClass,
late: bool,
expr: Option<P<Expr>>,
},
InOut {
reg: InlineAsmRegOrRegClass,
late: bool,
expr: P<Expr>,
},
SplitInOut {
reg: InlineAsmRegOrRegClass,
late: bool,
in_expr: P<Expr>,
out_expr: Option<P<Expr>>,
},
Const {
expr: P<Expr>,
},
Sym {
expr: P<Expr>,
},
}
```
The `asm!` macro is implemented in librustc_builtin_macros and outputs an `InlineAsm` AST node. The template string is parsed using libfmt_macros, positional and named operands are resolved to explicit operand indicies. Since target information is not available to macro invocations, validation of the registers and register classes is deferred to AST lowering.
## HIR
`InlineAsm` is represented as an expression in the HIR:
```rust
pub struct InlineAsm<'hir> {
pub template: &'hir [InlineAsmTemplatePiece],
pub operands: &'hir [InlineAsmOperand<'hir>],
pub options: InlineAsmOptions,
}
pub enum InlineAsmRegOrRegClass {
Reg(InlineAsmReg),
RegClass(InlineAsmRegClass),
}
pub enum InlineAsmOperand<'hir> {
In {
reg: InlineAsmRegOrRegClass,
expr: Expr<'hir>,
},
Out {
reg: InlineAsmRegOrRegClass,
late: bool,
expr: Option<Expr<'hir>>,
},
InOut {
reg: InlineAsmRegOrRegClass,
late: bool,
expr: Expr<'hir>,
},
SplitInOut {
reg: InlineAsmRegOrRegClass,
late: bool,
in_expr: Expr<'hir>,
out_expr: Option<Expr<'hir>>,
},
Const {
expr: Expr<'hir>,
},
Sym {
expr: Expr<'hir>,
},
}
```
AST lowering is where `InlineAsmRegOrRegClass` is converted from `Symbol`s to an actual register or register class. If any modifiers are specified for a template string placeholder, these are validated against the set allowed for that operand type. Finally, explicit registers for inputs and outputs are checked for conflicts (same register used for different operands).
## Type checking
Each register class has a whitelist of types that it may be used with. After the types of all operands have been determined, the `intrinsicck` pass will check that these types are in the whitelist. It also checks that split `inout` operands have compatible types and that `const` operands are integers or floats. Suggestions are emitted where needed if a template modifier should be used for an operand based on the type that was passed into it.
## HAIR
`InlineAsm` is represented as an expression in the HAIR:
```rust
crate enum ExprKind<'tcx> {
// [..]
InlineAsm {
template: &'tcx [InlineAsmTemplatePiece],
operands: Vec<InlineAsmOperand<'tcx>>,
options: InlineAsmOptions,
},
}
crate enum InlineAsmOperand<'tcx> {
In {
reg: InlineAsmRegOrRegClass,
expr: ExprRef<'tcx>,
},
Out {
reg: InlineAsmRegOrRegClass,
late: bool,
expr: Option<ExprRef<'tcx>>,
},
InOut {
reg: InlineAsmRegOrRegClass,
late: bool,
expr: ExprRef<'tcx>,
},
SplitInOut {
reg: InlineAsmRegOrRegClass,
late: bool,
in_expr: ExprRef<'tcx>,
out_expr: Option<ExprRef<'tcx>>,
},
Const {
expr: ExprRef<'tcx>,
},
SymFn {
expr: ExprRef<'tcx>,
},
SymStatic {
expr: ExprRef<'tcx>,
},
}
```
The only significant change compared to HIR is that `Sym` has been lowered to either a `SymFn` whose `expr` is a `Literal` ZST of the `fn`, or a `SymStatic` whose `expr` is a `StaticRef`.
## MIR
`InlineAsm` is represented as a `Terminator` in the MIR:
```rust
pub enum TerminatorKind<'tcx> {
// [..]
/// Block ends with an inline assembly block. This is a terminator since
/// inline assembly is allowed to diverge.
InlineAsm {
/// The template for the inline assembly, with placeholders.
template: &'tcx [InlineAsmTemplatePiece],
/// The operands for the inline assembly, as `Operand`s or `Place`s.
operands: Vec<InlineAsmOperand<'tcx>>,
/// Miscellaneous options for the inline assembly.
options: InlineAsmOptions,
/// Destination block after the inline assembly returns, unless it is
/// diverging (InlineAsmOptions::NORETURN).
destination: Option<BasicBlock>,
},
}
pub enum InlineAsmOperand<'tcx> {
In {
reg: InlineAsmRegOrRegClass,
value: Operand<'tcx>,
},
Out {
reg: InlineAsmRegOrRegClass,
late: bool,
place: Option<Place<'tcx>>,
},
InOut {
reg: InlineAsmRegOrRegClass,
late: bool,
in_value: Operand<'tcx>,
out_place: Option<Place<'tcx>>,
},
Const {
value: Operand<'tcx>,
},
SymFn {
value: Box<Constant<'tcx>>,
},
SymStatic {
value: Box<Constant<'tcx>>,
},
}
```
As part of HAIR lowering, `InOut` and `SplitInOut` operands are lowered to a split form with a separate `in_value` and `out_place`.
Semantically, the `InlineAsm` terminator is similar to the `Call` terminator except that it has multiple output places where a `Call` only has a single return place output.
The constant promotion pass is used to ensure that `const` operands are actually constants (using the same logic as `#[rustc_args_required_const]`).
## Codegen
Operands are lowered one more time before being passed to LLVM codegen:
```rust
pub enum InlineAsmOperandRef<'tcx, B: BackendTypes + ?Sized> {
In {
reg: InlineAsmRegOrRegClass,
value: OperandRef<'tcx, B::Value>,
},
Out {
reg: InlineAsmRegOrRegClass,
late: bool,
place: Option<PlaceRef<'tcx, B::Value>>,
},
InOut {
reg: InlineAsmRegOrRegClass,
late: bool,
in_value: OperandRef<'tcx, B::Value>,
out_place: Option<PlaceRef<'tcx, B::Value>>,
},
Const {
string: String,
},
SymFn {
instance: Instance<'tcx>,
},
SymStatic {
def_id: DefId,
},
}
```
The operands are lowered to LLVM operands and constraint codes as follow:
- `out` and the output part of `inout` operands are added first, as required by LLVM. Late output operands have a `=` prefix added to their constraint code, non-late output operands have a `=&` prefix added to their constraint code.
- `in` operands are added normally.
- `inout` operands are tied to the matching output operand.
- `sym` operands are passed as function pointers or pointers, using the `"s"` constraint.
- `const` operands are formatted to a string and directly inserted in the template string.
The template string is converted to LLVM form:
- `$` characters are escaped as `$$`.
- `const` operands are converted to strings and inserted directly.
- Placeholders are formatted as `${X:M}` where `X` is the operand index and `M` is the modifier character. Modifiers are converted from the Rust form to the LLVM form.
The various options are converted to clobber constraints or LLVM attributes, refer to the [RFC](https://github.com/Amanieu/rfcs/blob/inline-asm/text/0000-inline-asm.md#mapping-to-llvm-ir) for more details.
Note that LLVM is sometimes rather picky about what types it accepts for certain constraint codes so we sometimes need to insert conversions to/from a supported type. See the target-specific ISelLowering.cpp files in LLVM for details.
# Adding support for new architectures
Adding inline assembly support to an architecture is mostly a matter of defining the registers and register classes for that architecture. All the definitions for register classes are located in `src/librustc_target/asm/`.
Additionally you will need to implement lowering of these register classes to LLVM constraint codes in `src/librustc_codegen_llvm/asm.rs`.
Ignore arguments when looking for `IndexMut` for subsequent `mut` obligation
Given code like `v[&field].boo();` where `field: String` and
`.boo(&mut self)`, typeck will have decided that `v` is accessed using
`Index`, but when `boo` adds a new `mut` obligation,
`convert_place_op_to_mutable` is called. When this happens, for *some
reason* the arguments' dereference adjustments are completely ignored
causing an error saying that `IndexMut` is not satisfied:
```
error[E0596]: cannot borrow data in an index of `Indexable` as mutable
--> src/main.rs:30:5
|
30 | v[&field].boo();
| ^^^^^^^^^ cannot borrow as mutable
|
= help: trait `IndexMut` is required to modify indexed content, but it is not implemented for `Indexable`
```
This is not true, but by changing `try_overloaded_place_op` to retry
when given `Needs::MutPlace` without passing the argument types, the
example successfully compiles.
I believe there might be more appropriate ways to deal with this.
Fix#72002.
correctly handle uninferred consts
fixes the ICE mentioned in https://github.com/rust-lang/rust/issues/70507#issuecomment-615268893
I originally tried to generalize `need_type_info_err` to also work with consts which was not as much fun as I hoped 😅
It might be easier to have some duplication here and handle consts separately.
r? @varkor
Stabilize saturating_abs and saturating_neg
Stabilizes the following signed integer functions with saturation mechanics:
* saturating_abs()
* saturating_neg()
Closes#59983
impl From<Cow> for Box, Rc, and Arc
These forward `Borrowed`/`Owned` values to existing `From` impls.
- `Box<T>` is a fundamental type, so it would be a breaking change to add a blanket impl. Therefore, `From<Cow>` is only implemented for `[T]`, `str`, `CStr`, `OsStr`, and `Path`.
- For `Rc<T>` and `Arc<T>`, `From<Cow>` is implemented for everything that implements `From` the borrowed and owned types separately.
Lazy normalization of constants (Reprise)
Continuation of #67890 by @skinny121.
Initial implementation of #60471 for constants.
Perform normalization/evaluation of constants lazily, which is known as lazy normalization. Lazy normalization is only enabled when using `#![feature(lazy_normalization_consts)]`, by default constants are still evaluated eagerly as there are currently.
Lazy normalization of constants is achieved with a new ConstEquate predicate which type inferences uses to delay checking whether constants are equal to each other until later, avoiding cycle errors.
Note this doesn't allow the use of generics within repeat count expressions as that is still evaluated during conversion to mir. There are also quite a few other known problems with lazy normalization which will be fixed in future PRs.
r? @nikomatsakis
fixes#71922, fixes#71986
Support coercion between (FnDef | Closure) and (FnDef | Closure)
Fixes#46742, fixes#48109
Inject `Closure` into the `FnDef x FnDef` coercion special case, which makes coercion of `(FnDef | Closure) x (FnDef | Closure)` possible, where closures should be **non-capturing**.
`fold` is currently implemented via `try_fold`, but implementing it
directly results in slightly less LLVM IR being generated, speeding up
compilation of some benchmarks.
(And likewise for `rfold`.)
The commit adds `fold` implementations to all the iterators that lack
one but do have a `try_fold` implementation. Most of these just call the
`try_fold` implementation directly.
Disallow forbidden usage of non-ascii identifiers.
Part of RFC2457, this tightens allowed identifiers back to ascii only in two situations.
r? @petrochenkov
