or-patterns: Uniformly use `PatKind::Or` in AST & Fix/Cleanup resolve
Following up on work in https://github.com/rust-lang/rust/pull/63693 and https://github.com/rust-lang/rust/pull/61708, in this PR we:
- Uniformly use `PatKind::Or(...)` in AST:
- Change `ast::Arm.pats: Vec<P<Pat>>` => `ast::Arm.pat: P<Pat>`
- Change `ast::ExprKind::Let.0: Vec<P<Pat>>` => `ast::ExprKind::Let.0: P<Pat>`
- Adjust `librustc_resolve/late.rs` to correctly handle or-patterns at any level of nesting as a result.
In particular, the already-bound check which rejects e.g. `let (a, a);` now accounts for or-patterns. The consistency checking (ensures no missing bindings and binding mode consistency) also now accounts for or-patterns. In the process, a bug was found in the current compiler which allowed:
```rust
enum E<T> { A(T, T), B(T) }
use E::*;
fn foo() {
match A(0, 1) {
B(mut a) | A(mut a, mut a) => {}
}
}
```
The new algorithms took a few iterations to get right. I tried several clever schemes but ultimately a version based on a stack of hashsets and recording product/sum contexts was chosen since it is more clearly correct.
- Clean up `librustc_resolve/late.rs` by, among other things, using a new `with_rib` function to better ensure stack dicipline.
- Do not push the change in AST to HIR for now to avoid doing too much in this PR. To cope with this, we introduce a temporary hack in `rustc::hir::lowering` (clearly marked in the diff).
cc https://github.com/rust-lang/rust/issues/54883
cc @dlrobertson @matthewjasper
r? @petrochenkov
Use unicode-xid crate instead of libcore
This PR proposes to remove `char::is_xid_start` and `char::is_xid_continue` functions from `libcore` and use `unicode_xid` crate from crates.io (note that this crate is already present in rust-lang/rust's Cargo.lock).
Reasons to do this:
* removing rustc-binary-specific stuff from libcore
* making sure that, across the ecosystem, there's a single definition of what rust identifier is (`unicode-xid` has almost 10 million downs, as a `proc_macro2` dependency)
* making it easier to share `rustc_lexer` crate with rust-analyzer: no need to `#[cfg]` if we are building as a part of the compiler
Reasons not to do this:
* increased maintenance burden: we'll need to upgrade unicode version both in libcore and in unicode-xid. However, this shouldn't be a too heavy burden: just running `./unicode.py` after new unicode version. I (@matklad) am ready to be a t-compiler side maintainer of unicode-xid. Moreover, given that xid-unicode is an important dependency of syn, *someone* needs to maintain it anyway.
* xid-unicode implementation is significantly slower. It uses a more compact table with binary search, instead of a trie. However, this shouldn't matter in practice, because we have fast-path for ascii anyway, and code size savings is a plus. Moreover, in #59706 not using libcore turned out to be *faster*, presumably beacause checking for whitespace with match is even faster.
<details>
<summary>old description</summary>
Followup to #59706
r? @eddyb
Note that this doesn't actually remove tables from libcore, to avoid conflict with https://github.com/rust-lang/rust/pull/62641.
cc https://github.com/unicode-rs/unicode-xid/pull/11
</details>
They are only used by rustc_lexer, and are not needed elsewhere.
So we move the relevant definitions into rustc_lexer (while the actual
unicode data comes from the unicode-xid crate) and make the rest of
the compiler use it.
This allows lints and other diagnostics to refer to items
by a unique ID instead of relying on whacky path
resolution schemes that may break when items are
relocated.
resolve: Block expansion of a derive container until all its derives are resolved
So, it turns out there's one more reason to block expansion of a `#[derive]` container until all the derives inside it are resolved, beside `Copy` (https://github.com/rust-lang/rust/pull/63248).
The set of derive helper attributes registered by derives in the container also has to be known before the derives themselves are expanded, otherwise it may be too late (see https://github.com/rust-lang/rust/pull/63468#issuecomment-524550872 and the `#[stable_hasher]`-related test failures in https://github.com/rust-lang/rust/pull/63468).
So, we stop our attempts to unblock the container earlier, as soon as the `Copy` status is known, and just block until all its derives are resolved.
After all the derives are resolved we immediately go and process their helper attributes in the item, without delaying it until expansion of the individual derives.
Unblocks https://github.com/rust-lang/rust/pull/63468
r? @matthewjasper (as a reviewer of https://github.com/rust-lang/rust/pull/63248)
cc @c410-f3r
Recover `mut $pat` and other improvements
- Recover on e.g. `mut Foo(x, y)` and suggest `Foo(mut x, mut y)`. Fixes https://github.com/rust-lang/rust/issues/63764.
- Recover on e.g. `let mut mut x;`
- Recover on e.g. `let keyword` and `let keyword(...)`.
- Cleanups in `token.rs` with `fn is_non_raw_ident_where` and friends.
Propagate spans and attributes from proc macro definitions
Thanks to https://github.com/rust-lang/rust/pull/63269 we now have spans and attributes from proc macro definitions available in metadata.
However, that PR didn't actually put them into use! This PR finishes that work.
Attributes `rustc_macro_transparency`, `allow_internal_unstable`, `allow_internal_unsafe`, `local_inner_macros`, `rustc_builtin_macro`, `stable`, `unstable`, `rustc_deprecated`, `deprecated` now have effect when applied to proc macro definition functions.
From those attributes only `deprecated` is both stable and supposed to be used in new code.
(`#![staged_api]` still cannot be used in proc macro crates for unrelated reasons though.)
`Span::def_site` from the proc macro API now returns the correct location of the proc macro definition.
Also, I made a mistake in https://github.com/rust-lang/rust/pull/63269#discussion_r312702919, loaded proc macros didn't actually use the resolver cache.
This PR fixes the caching issue, now proc macros go through the `Resolver::macro_map` cache as well.
(Also, the first commit turns `proc_macro::quote` into a regular built-in macro to reduce the number of places where `SyntaxExtension`s need to be manually created.)
Refactor feature gates
After #63824, this goes a few steps further by
- parsing doc comments in the macros to extract descriptions for feature gates, and
- introducing a common `Feature` type to replace the tuples used previously to improve readability.
The descriptions are not yet used, but I felt like this PR is a useful enough refactoring on its own.
r? @Centril
Fully implement or-pattern parsing
Builds upon the initial parsing in https://github.com/rust-lang/rust/pull/61708 to fully implement or-pattern (`p | q`) parsing as specified in [the grammar section of RFC 2535](https://github.com/rust-lang/rfcs/blob/master/text/2535-or-patterns.md#grammar).
Noteworthy:
- We allow or-patterns in `[p | q, ...]`.
- We allow or-patterns in `let` statements and `for` expressions including with leading `|`.
- We improve recovery for `p || q` (+ tests for that in `multiple-pattern-typo.rs`).
- We improve recovery for `| p | q` in inner patterns (tests in `or-patterns-syntactic-fail.rs`).
- We rigorously test or-pattern parsing (in `or-patterns-syntactic-{pass,fail}.rs`).
- We harden the feature gating tests.
- We do **_not_** change `ast.rs`. That is, `ExprKind::Let.0` and `Arm.pats` still accept `Vec<P<Pat>>`.
I was starting work on that but it would be cleaner to do this in a separate PR so this one has a narrower scope.
cc @dlrobertson
cc the tracking issue https://github.com/rust-lang/rust/issues/54883.
r? @estebank
Also mark derive helpers as known as a part of the derive container's expansion instead of expansion of the derives themselves which may happen too late.
This replaces the ad-hoc tuples used in the different feature gate files
and unifies their content into a common type, leading to more readable
matches and other good stuff that comes from having named fields. It
also contains the description of each feature as extracted from the doc
comment.
