The idea here is to make a reusable library out of the existing
rust-lexer, by separating out pure lexing and rustc-specific concerns,
like spans, error reporting an interning.
So, rustc_lexer operates directly on `&str`, produces simple tokens
which are a pair of type-tag and a bit of original text, and does not
report errors, instead storing them as flags on the token.
Identify when a stmt could have been parsed as an expr
There are some expressions that can be parsed as a statement without
a trailing semicolon depending on the context, which can lead to
confusing errors due to the same looking code being accepted in some
places and not others. Identify these cases and suggest enclosing in
parenthesis making the parse non-ambiguous without changing the
accepted grammar.
Fix#54186, cc #54482, fix#59975, fix#47287.
Adds support for .await under the existing async_await feature gate.
Moves macro-like await! syntax to the await_macro feature gate.
Removes support for `await` as a non-keyword under the `async_await`
feature.
There are some expressions that can be parsed as a statement without
a trailing semicolon depending on the context, which can lead to
confusing errors due to the same looking code being accepted in some
places and not others. Identify these cases and suggest enclosing in
parenthesis making the parse non-ambiguous without changing the
accepted grammar.
This commit changes `syntax::fold::Folder` from a functional style
(where most methods take a `T` and produce a new `T`) to a more
imperative style (where most methods take and modify a `&mut T`), and
renames it `syntax::mut_visit::MutVisitor`.
The first benefit is speed. The functional style does not require any
reallocations, due to the use of `P::map` and
`MoveMap::move_{,flat_}map`. However, every field in the AST must be
overwritten; even those fields that are unchanged are overwritten with
the same value. This causes a lot of unnecessary memory writes. The
imperative style reduces instruction counts by 1--3% across a wide range
of workloads, particularly incremental workloads.
The second benefit is conciseness; the imperative style is usually more
concise. E.g. compare the old functional style:
```
fn fold_abc(&mut self, abc: ABC) {
ABC {
a: fold_a(abc.a),
b: fold_b(abc.b),
c: abc.c,
}
}
```
with the imperative style:
```
fn visit_abc(&mut self, ABC { a, b, c: _ }: &mut ABC) {
visit_a(a);
visit_b(b);
}
```
(The reductions get larger in more complex examples.)
Overall, the patch removes over 200 lines of code -- even though the new
code has more comments -- and a lot of the remaining lines have fewer
characters.
Some notes:
- The old style used methods called `fold_*`. The new style mostly uses
methods called `visit_*`, but there are a few methods that map a `T`
to something other than a `T`, which are called `flat_map_*` (`T` maps
to multiple `T`s) or `filter_map_*` (`T` maps to 0 or 1 `T`s).
- `move_map.rs`/`MoveMap`/`move_map`/`move_flat_map` are renamed
`map_in_place.rs`/`MapInPlace`/`map_in_place`/`flat_map_in_place` to
reflect their slightly changed signatures.
- Although this commit renames the `fold` module as `mut_visit`, it
keeps it in the `fold.rs` file, so as not to confuse git. The next
commit will rename the file.
Replace usages of `..i + 1` ranges with `..=i`.
Before this change we were using old computer code techniques. After this change we use the new and improved computer code techniques.
There were only some UI tests that covered this function.
Since there's more diagnostic work going on, I think it makes
sense to have this unit tested.
TokenStream::extend
Two new insta-stable impls in libproc_macro:
```rust
impl Extend<TokenTree> for TokenStream
impl Extend<TokenStream> for TokenStream
```
`proc_macro::TokenStream` already implements `FromIterator<TokenTree>` and `FromIterator<TokenStream>` so I elected to support the same input types for `Extend`.
**This commit reduces compile time of Serde derives by 60% (takes less than half as long to compile)** as measured by building our test suite:
```console
$ git clone https://github.com/serde-rs/serde
$ cd serde/test_suite
$ cargo check --tests --features proc-macro2/nightly
$ rm -f ../target/debug/deps/libtest_*.rmeta
$ time cargo check --tests --features proc-macro2/nightly
Before: 20.8 seconds
After: 8.6 seconds
```
r? @alexcrichton
