The idea here is to make a reusable library out of the existing
rust-lexer, by separating out pure lexing and rustc-specific concerns,
like spans, error reporting an interning.
So, rustc_lexer operates directly on `&str`, produces simple tokens
which are a pair of type-tag and a bit of original text, and does not
report errors, instead storing them as flags on the token.
Cancel unemitted diagnostics during error recovery
Follow up to https://github.com/rust-lang/rust/pull/62604. Use @eddyb's preferred style and catch other case of the same problem.
r? @eddyb
Fix#62660
If the explicitly given type of a `self` parameter fails to parse correctly, we need to propagate the error rather than dropping it and causing an ICE.
Fixes#62660.
Do not emit type errors after parse error in last statement of block
When recovering from a parse error inside a block, do not emit type
errors generating on that block's recovered return expression.
Fix#57383.
If the explicitly given type of a `self` parameter fails to parse correctly,
we need to propagate the error rather than dropping it and causing an ICE.
Fixes#62660.
Remove support for 1-token lookahead from the lexer
`StringReader` maintained `peek_token` and `peek_span_src_raw` for look ahead.
`peek_token` was used only by rustdoc syntax coloring. After moving peeking logic into highlighter, I was able to remove `peek_token` from the lexer. I tried to use `iter::Peekable`, but that wasn't as pretty as I hoped, due to buffered fatal errors. So I went with hand-rolled peeking.
After that I've noticed that the only peeking behavior left was for raw tokens to test tt jointness. I've rewritten it in terms of trivia tokens, and not just spans.
After that it became possible to simplify the awkward constructor of the lexer, which could return `Err` if the first peeked token contained error.
Remove io::Result from syntax::print
Since we're now writing directly to the vector, there's no need to
thread results through the whole printing infrastructure
The reader itself doesn't need ability to peek tokens, so it's better
if clients implement this functionality.
This hopefully becomes especially easy once we use iterator interface
for lexer, but this is not too easy at the moment, because of buffered
errors.
