Simplify `TokenStream` some more
These commits simplify `TokenStream`, remove `ThinTokenStream`, and avoid some clones. The end result is simpler code and a slight perf win on some benchmarks.
r? @petrochenkov
Implement basic input validation for built-in attributes
Correct top-level shape (`#[attr]` vs `#[attr(...)]` vs `#[attr = ...]`) is enforced for built-in attributes, built-in attributes must also fit into the "meta-item" syntax (aka the "classic attribute syntax").
For some subset of attributes (found by crater run), errors are lowered to deprecation warnings.
NOTE: This PR previously included https://github.com/rust-lang/rust/pull/57367 as well.
`TokenStream` is currently recursive in *two* ways:
- the `TokenTree` variant contains a `ThinTokenStream`, which can
contain a `TokenStream`;
- the `TokenStream` variant contains a `Vec<TokenStream>`.
The latter is not necessary and causes significant complexity. This
commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`.
This reduces complexity significantly. In particular, `StreamCursor` is
eliminated, and `Cursor` becomes much simpler, consisting now of just a
`TokenStream` and an index.
The commit also removes the `Extend` impl for `TokenStream`, because it
is only used in tests. (The commit also removes those tests.)
Overall, the commit reduces the number of lines of code by almost 200.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
resolve: Implement prelude search for macro paths, implement tool attributes
When identifier is macro path is resolved in scopes (i.e. the first path segment - `foo` in `foo::mac!()` or `foo!()`), scopes are searched in the same order as for non-macro paths - items in modules, extern prelude, tool prelude (see later), standard library prelude, language prelude, but with some extra shadowing restrictions (names from globs and macro expansions cannot shadow names from outer scopes). See the comment in `fn resolve_lexical_macro_path_segment` for more details.
"Tool prelude" currently contains two "tool modules" `rustfmt` and `clippy`, and is searched immediately after extern prelude.
This makes the [possible long-term solution](https://github.com/rust-lang/rfcs/blob/master/text/2103-tool-attributes.md#long-term-solution) for tool attributes exactly equivalent to the existing extern prelude scheme, except that `--extern=my_crate` making crate names available in scope is replaced with something like `--tool=my_tool` making tool names available in scope.
The `tool_attributes` feature is still unstable and `#![feature(tool_attributes)]` now implicitly enables `#![feature(use_extern_macros)]`. `use_extern_macros` is a prerequisite for `tool_attributes`, so their stabilization will happen in the same order.
If `use_extern_macros` is not enabled, then tool attributes are treated as custom attributes (this is temporary, anyway).
Fixes https://github.com/rust-lang/rust/issues/52576
Fixes https://github.com/rust-lang/rust/issues/52512
Fixes https://github.com/rust-lang/rust/issues/51277
cc https://github.com/rust-lang/rust/issues/52269
