Remove `TokenStream::JointTree`.
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
Fix stabilization version numbers (exhaustive_integer_patterns + macro_literal_matcher)
+ `exhaustive_integer_patterns` slipped 1.32; merged in 1.33 -- https://github.com/rust-lang/rust/pull/56362
+ `macro_literal_matcher` isn't stable on current stable (1.31) but is on beta (1.32).
r? @varkor
Rework treatment of `$crate` in procedural macros
Important clarification: `$crate` below means "processed `$crate`" or "output `$crate`". In the input of a decl macro `$crate` is just two separate tokens, but in the *output of a decl macro* `$crate` is a single keyword identifier (https://github.com/rust-lang/rust/issues/55640#issuecomment-435692791).
First of all, this PR removes the `eliminate_crate_var` hack.
`$crate::foo` is no longer replaced with `::foo` or `::crate_name::foo` in the input of derive proc macros, it's passed to the macro instead with its precise span and hygiene data, and can be treated as any other path segment keyword (like `crate` or `self`) after that. (Note: `eliminate_crate_var` was never used for non-derive proc macros.)
This creates an annoying problem - derive macros still may stringify their input before processing and expect `$crate` survive that stringification and refer to the same crate (the Rust 1.15-1.29 way of doing things).
Moreover, the input of proc macro attributes and derives (but not fn-like proc macros) also effectively survives stringification before being passed to the macro (also for legacy implementation reasons).
So we kind of resurrect the `eliminate_crate_var` hack in reduced form, but apply it only to AST pretty-printing.
If an AST fragment is pretty-printed, the resulting *text* will have `$crate` replaced with `crate` or `::crate_name`. This should be enough to keep all the legacy cases working.
Closes https://github.com/rust-lang/rust/issues/55640
Closes https://github.com/rust-lang/rust/issues/56622
r? @ghost
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
overhaul external doc attribute diagnostics
This PR improves the error handling and spans for the external doc attribute. Many cases that silently failed before now emit errors, spans are tightened, and the errors have help and suggestions.
I tried to address all the cases that users ran into in the tracking issue.
cc #44732
r? @QuietMisdreavus
2018 edition - confusing error message when declaring unnamed parameters
Fixes#53990.
This PR adds a note providing context for the change to argument
names being required in the 2018 edition for trait methods and a
suggestion for the fix.
Use a `newtype_index!` within `Symbol`.
This shrinks `Option<Symbol>` from 8 bytes to 4 bytes, which shrinks
`Token` from 24 bytes to 16 bytes. This reduces instruction counts by up
to 1% across a range of benchmarks.
r? @oli-obk
Add non-panicking `maybe_new_parser_from_file` variant
Add (seemingly?) missing `maybe_new_parser_from_file` constructor variant.
Disclaimer: I'm not certain this is the correct approach - just found out we don't have this when working on a Rustfmt PR to catch/prevent more Rust parser panics: https://github.com/rust-lang/rustfmt/pull/3240 and tried to make it work somehow.
* Update bootstrap compiler
* Update version to 1.33.0
* Remove some `#[cfg(stage0)]` annotations
Actually updating the version number is blocked on updating Cargo
This shrinks:
- ThinTokenStream: 16 to 8 bytes
- TokenTree: 32 to 24 bytes
- TokenStream: 40 to 32 bytes
The only downside is that in a couple of places this requires using
`to_vec()` (which allocates) instead of `sub_slice()`. But those places
are rarely executed, so it doesn't hurt perf.
Overall, this reduces instruction counts on numerous benchmarks by up to
3%.
This shrinks `Option<Symbol>` from 8 bytes to 4 bytes, which shrinks
`Token` from 24 bytes to 16 bytes. This reduces instruction counts by up
to 1% across a range of benchmarks.
Remove `tokenstream::Delimited`.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
This commit replaces many usages of `File::open` and reading or writing
with `fs::read_to_string`, `fs::read` and `fs::write`. This reduces code
complexity, and will improve performance for most reads, since the
functions allocate the buffer to be the size of the file.
I believe that this commit will not impact behavior in any way, so some
matches will check the error kind in case the file was not valid UTF-8.
Some of these cases may not actually care about the error.
Replace usages of `..i + 1` ranges with `..=i`.
Before this change we were using old computer code techniques. After this change we use the new and improved computer code techniques.
