Simplify `TokenStream` some more
These commits simplify `TokenStream`, remove `ThinTokenStream`, and avoid some clones. The end result is simpler code and a slight perf win on some benchmarks.
r? @petrochenkov
Implement basic input validation for built-in attributes
Correct top-level shape (`#[attr]` vs `#[attr(...)]` vs `#[attr = ...]`) is enforced for built-in attributes, built-in attributes must also fit into the "meta-item" syntax (aka the "classic attribute syntax").
For some subset of attributes (found by crater run), errors are lowered to deprecation warnings.
NOTE: This PR previously included https://github.com/rust-lang/rust/pull/57367 as well.
Use structured suggestions for nonstandard style lints
This PR modifies the lints in the nonstandard_style group to use structured suggestions. Note that there's a bit of tricky span calculation going on for the `crate_name` attribute. It also simplifies the code a bit: I don't think the "fallback" suggestions for these lints can actually be triggered.
Fixes#48103.
Fixes#52414.
`TokenStream::Stream` can represent a token stream containing any number
of token trees. `TokenStream::Tree` is the special case representing a
single token tree. The latter doesn't occur all that often dynamically,
so this commit removes it, which simplifies the code quite a bit.
This change has mixed performance effects.
- The size of `TokenStream` drops from 32 bytes to 8 bytes, and there
is one less case for all the match statements.
- The conversion of a `TokenTree` to a `TokenStream` now requires two
allocations, for the creation of a single element Lrc<Vec<_>>. (But a
subsequent commit in this PR will reduce the main source of such
conversions.)
Make `TokenStream` less recursive.
`TokenStream` is currently recursive in *two* ways:
- the `TokenTree` variant contains a `ThinTokenStream`, which can
contain a `TokenStream`;
- the `TokenStream` variant contains a `Vec<TokenStream>`.
The latter is not necessary and causes significant complexity. This
commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`.
This reduces complexity significantly. In particular, `StreamCursor` is
eliminated, and `Cursor` becomes much simpler, consisting now of just a
`TokenStream` and an index.
The commit also removes the `Extend` impl for `TokenStream`, because it
is only used in tests. (The commit also removes those tests.)
Overall, the commit reduces the number of lines of code by almost 200.
`TokenStream` is currently recursive in *two* ways:
- the `TokenTree` variant contains a `ThinTokenStream`, which can
contain a `TokenStream`;
- the `TokenStream` variant contains a `Vec<TokenStream>`.
The latter is not necessary and causes significant complexity. This
commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`.
This reduces complexity significantly. In particular, `StreamCursor` is
eliminated, and `Cursor` becomes much simpler, consisting now of just a
`TokenStream` and an index.
The commit also removes the `Extend` impl for `TokenStream`, because it
is only used in tests. (The commit also removes those tests.)
Overall, the commit reduces the number of lines of code by almost 200.
This commit completely removes usage of the `panictry!` macro from
outside libsyntax. The macro causes parse errors to be fatal, so using
it in libsyntax_ext caused parse failures *within* a syntax extension to
be fatal, which is probably not intended.
Furthermore, this commit adds spans to diagnostics emitted by empty
extensions if they were missing, à la #56491.
When a format string has escaped whitespace characters format
arguments were shifted by one per each escaped character. Account
for these escaped characters when synthesizing the spans.
Fix#55155.
- Point at opening mismatched formatting brace
- Account for differences between raw and regular strings
- Account for differences between the code snippet and `InternedString`
- Add more tests
Remove `TokenStream::JointTree`.
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
Remove `tokenstream::Delimited`.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
