syntax-[breaking-change] cc #31645
(Only breaking because ast::TokenTree is now tokenstream::TokenTree.)
This pull request refactors TokenTrees into their own file as src/libsyntax/tokenstream.rs, moving them out of src/libsyntax/ast.rs, in order to prepare for an accompanying TokenStream implementation (per RFC 1566).
This PR refactors the 'errors' part of libsyntax into its own crate (librustc_errors). This is the first part of a few refactorings to simplify error reporting and potentially support more output formats (like a standardized JSON output and possibly an --explain mode that can work with the user's code), though this PR stands on its own and doesn't assume further changes.
As part of separating out the errors crate, I have also refactored the code position portion of codemap into its own crate (libsyntax_pos). While it's helpful to have the common code positions in a separate crate for the new errors crate, this may also enable further simplifications in the future.
**syntax-[breaking-change]** cc #31645
New `TraitItemKind::Macro` variant
This change adds support for macro expansion inside trait items by adding the new `TraitItemKind::Macro` and associated parsing code.
When items are inlined from extern crates, the filename in the debug info
is taken from the FileMap that's serialized in the rlib metadata.
Currently this is just FileMap.name, which is whatever path is passed to rustc.
Since libcore and libstd are built by invoking rustc with relative paths,
they wind up with relative paths in the rlib, and when linked into a binary
the debug info uses relative paths for the names, but since the compilation
directory for the final binary, tools trying to read source filenames
will wind up with bad paths. We noticed this in Firefox with source
filenames from libcore/libstd having bad paths.
This change stores an absolute path in FileMap.abs_path, and uses that
if available for writing debug info. This is not going to magically make
debuggers able to find the source, but it will at least provide sensible
paths.
Casual grepping revealed some places in the codebase (some of which
antedated `if let`'s December 2014 stabilization in c200ae5a) where we
were using a match with a `None => ()` arm where (in the present
author's opinion) an `if let` conditional would be more readable. (Other
places where matching to the unit value did seem to better express the
intent were left alone.)
It's likely that we don't care about making such trivial,
non-functional, sheerly æsthetic changes.
But if we do, this is a patch.
This makes the "shadowing labels" warning *not* print the entire loop
as a span, but only the lifetime.
Also makes #31719 go away, but does not fix its root cause (the span
of the expanded loop is still wonky, but not used anymore).
Fix spans and expected token lists, fix#33413 + other cosmetic improvements
Add test for #33413
Convert between `Arg` and `ExplicitSelf` precisely
Simplify pretty-printing for methods
parser: do not try to continue with `unsafe` on foreign fns
The changed line makes it look like `unsafe` is allowed, but the first statement of `parse_item_foreign_fn` is:
```
self.expect_keyword(keywords::Fn)?;
```
So we get the strange "expected one of `fn`, `pub`, `static`, or `unsafe`, found `unsafe`".
Fixes: #27361
parser: fix suppression of syntax errors in range RHS
Invalid expressions on the RHS were just swallowed without generating an error. The new version more closely mirrors the code for parsing `..x` in the `parse_prefix_range_expr` method below, where no cancel is done either.
Fixes#33262.
Add more aliases for Unicode confusable chars
Building upon #29837, this PR:
* added aliases for space characters,
* distinguished square brackets from parens, and
* added common CJK punctuation characters as aliases.
This will especially help CJK users who may have forgotten to switch off IME when coding.
parser: change warning into an error on `T<A=B, C>`
part of #32214
This seems to be the obvious fix, and the error message is consistent with all the other parser errors ("expected x, found y").
lexer: do not display char confusingly in error message
Current code leads to messages like `... use a \xHH escape: \u{e4}` which is confusing.
The printed span already points to the offending character, which should be enough to identify the non-ASCII problem.
Fixes: #29088
The extra filename and line was mainly there to keep the indentation
relative to the main snippet; now that this doesn't include
filename/line-number as a prefix, it is distracted.
The changed line makes it look like `unsafe` is allowed, but the
first statement of `parse_item_foreign_fn` is:
`self.expect_keyword(keywords::Fn)?;`
So we get the strange "expected one of `fn`, `pub`, `static`, or
`unsafe`, found `unsafe`".
Fixes: #27361
Current code leads to messages like "... use a \xHH escape: \u{e4}"
which is confusing.
The printed span already points to the offending character, which
should be enough to identify the non-ASCII problem.
Fixes: #29088
Invalid expressions on the RHS were just swallowed without generating
an error. The new code more closely mirrors the code for parsing
`..x` in the `parse_prefix_range_expr` method, where no cancel is done
either.
Fixes#33262.
Paths are mostly parsed without taking whitespaces into account, e.g. `std :: vec :: Vec :: new ()` parses successfully, however, there are some special cases involving keywords `super`, `self` and `Self`. For example, `self::` is considered a path start only if there are no spaces between `self` and `::`. These restrictions probably made sense when `self` and friends weren't keywords, but now they are unnecessary.
The first two commits remove this special treatment of whitespaces by removing `token::IdentStyle` entirely and therefore fix https://github.com/rust-lang/rust/issues/14109.
This change also affects naked `self` and `super` (which are not tightly followed by `::`, obviously) they can now be parsed as paths, however they are still not resolved correctly in imports (cc @jseyfried, see `compile-fail/use-keyword.rs`), so https://github.com/rust-lang/rust/issues/29036 is not completely fixed.
The third commit also makes `super`, `self`, `Self` and `static` keywords nominally (before this they acted as keywords for all purposes) and removes most of remaining \"special idents\".
The last commit (before tests) contains some small improvements - some qualified paths with type parameters are parsed correctly, `parse_path` is not used for parsing single identifiers, imports are sanity checked for absence of type parameters - such type parameters can be generated by syntax extensions or by macros when https://github.com/rust-lang/rust/issues/10415 is fixed (~~soon!~~already!).
This patch changes some pretty basic things in `libsyntax`, like `token::Token` and the keyword list, so it's a plugin-[breaking-change].
r? @eddyb
