Remove `TokenStream::JointTree`.
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
Rework treatment of `$crate` in procedural macros
Important clarification: `$crate` below means "processed `$crate`" or "output `$crate`". In the input of a decl macro `$crate` is just two separate tokens, but in the *output of a decl macro* `$crate` is a single keyword identifier (https://github.com/rust-lang/rust/issues/55640#issuecomment-435692791).
First of all, this PR removes the `eliminate_crate_var` hack.
`$crate::foo` is no longer replaced with `::foo` or `::crate_name::foo` in the input of derive proc macros, it's passed to the macro instead with its precise span and hygiene data, and can be treated as any other path segment keyword (like `crate` or `self`) after that. (Note: `eliminate_crate_var` was never used for non-derive proc macros.)
This creates an annoying problem - derive macros still may stringify their input before processing and expect `$crate` survive that stringification and refer to the same crate (the Rust 1.15-1.29 way of doing things).
Moreover, the input of proc macro attributes and derives (but not fn-like proc macros) also effectively survives stringification before being passed to the macro (also for legacy implementation reasons).
So we kind of resurrect the `eliminate_crate_var` hack in reduced form, but apply it only to AST pretty-printing.
If an AST fragment is pretty-printed, the resulting *text* will have `$crate` replaced with `crate` or `::crate_name`. This should be enough to keep all the legacy cases working.
Closes https://github.com/rust-lang/rust/issues/55640
Closes https://github.com/rust-lang/rust/issues/56622
r? @ghost
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
2018 edition - confusing error message when declaring unnamed parameters
Fixes#53990.
This PR adds a note providing context for the change to argument
names being required in the 2018 edition for trait methods and a
suggestion for the fix.
Use a `newtype_index!` within `Symbol`.
This shrinks `Option<Symbol>` from 8 bytes to 4 bytes, which shrinks
`Token` from 24 bytes to 16 bytes. This reduces instruction counts by up
to 1% across a range of benchmarks.
r? @oli-obk
Add non-panicking `maybe_new_parser_from_file` variant
Add (seemingly?) missing `maybe_new_parser_from_file` constructor variant.
Disclaimer: I'm not certain this is the correct approach - just found out we don't have this when working on a Rustfmt PR to catch/prevent more Rust parser panics: https://github.com/rust-lang/rustfmt/pull/3240 and tried to make it work somehow.
This shrinks `Option<Symbol>` from 8 bytes to 4 bytes, which shrinks
`Token` from 24 bytes to 16 bytes. This reduces instruction counts by up
to 1% across a range of benchmarks.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
53956 panic on include bytes of own file
fix#53956
When using `include_bytes!` on a source file in the project, compiler would panic on subsequent compilations because `expand_include_bytes` would overwrite files in the source_map with no source. This PR changes `expand_include_bytes` to check source_map and use the already existing src, if any.
syntax: Use iterator and pattern APIs instead of `char_at`
Iterating over chars with the `char_at(str, i)` `i += ch.len_utf8()` loop seems unidiomatic.
`CrateRoot` -> `PathRoot`, `::` doesn't necessarily mean crate root now
`SelfValue` -> `SelfLower`, `SelfType` -> `SelfUpper`, both `self` and `Self` can be used in type and value namespaces now
Remove not used `DotEq` token
Currently libproc_macro does not use `DotEq` token.
https://github.com/rust-lang/rust/pull/49545 changed libproc_macro
to not generate `DotEq` token.
