Split libsyntax apart
In this PR the general idea is to separate the AST, parser, and friends by a more data / logic structure (tho not fully realized!) by separating out the parser and macro expansion code from libsyntax. Specifically have now three crates instead of one (libsyntax):
- libsyntax:
- concrete syntax tree (`syntax::ast`)
- definition of tokens and token-streams (`syntax::{token, tokenstream}`) -- used by `syntax::ast`
- visitors (`syntax::visit`, `syntax::mut_visit`)
- shared definitions between `libsyntax_expand`
- feature gating (`syntax::feature_gate`) -- we could possibly move this out to its own crater later.
- attribute and meta item utilities, including used-marking (`syntax::attr`)
- pretty printer (`syntax::print`) -- this should possibly be moved out later. For now I've reduced down the dependencies to a single essential one which could be broken via `ParseSess`. This entails that e.g. `Debug` impls for `Path` cannot reference the pretty printer.
- definition of `ParseSess` (`syntax::sess`) -- this is used by `syntax::{attr, print, feature_gate}` and is a common definition used by the parser and other things like librustc.
- the `syntax::source_map` -- this includes definitions used by `syntax::ast` and other things but could ostensibly be moved `syntax_pos` since that is more related to this module.
- a smattering of misc utilities not sufficiently important to itemize -- some of these could be moved to where they are used (often a single place) but I wanted to limit the scope of this PR.
- librustc_parse:
- parser (`rustc_parse::parser`) -- reading a file and such are defined in the crate root tho.
- lexer (`rustc_parse::lexer`)
- validation of meta grammar (post-expansion) in (`rustc_parse::validate_attr`)
- libsyntax_expand -- this defines the infra for macro expansion and conditional compilation but this is not libsyntax_ext; we might want to merge them later but currently libsyntax_expand is depended on by librustc_metadata which libsyntax_ext is not.
- conditional compilation (`syntax_expand::config`) -- moved from `syntax::config` to here
- the bulk of this crate is made up of the old `syntax::ext`
r? @estebank
Point at formatting descriptor string when it is invalid
When a formatting string contains an invalid descriptor, point at it
instead of the argument:
```
error: unknown format trait `foo`
--> $DIR/ifmt-bad-arg.rs:86:17
|
LL | println!("{:foo}", 1);
| ^^^
|
= note: the only appropriate formatting traits are:
- ``, which uses the `Display` trait
- `?`, which uses the `Debug` trait
- `e`, which uses the `LowerExp` trait
- `E`, which uses the `UpperExp` trait
- `o`, which uses the `Octal` trait
- `p`, which uses the `Pointer` trait
- `b`, which uses the `Binary` trait
- `x`, which uses the `LowerHex` trait
- `X`, which uses the `UpperHex` trait
```
We also sever syntax's dependency on rustc_target as a result.
This should slightly improve pipe-lining.
Moreover, some cleanup is done in related code.
`AttrKind` is a new type with two variants, `Normal` and `DocComment`. It's a
big performance win (over 10% in some cases) because `DocComment` lets doc
comments (which are common) be represented very cheaply.
`Attribute` gets some new helper methods to ease the transition:
- `has_name()`: check if the attribute name matches a single `Symbol`; for
`DocComment` variants it succeeds if the symbol is `sym::doc`.
- `is_doc_comment()`: check if it has a `DocComment` kind.
- `{get,unwrap}_normal_item()`: extract the item from a `Normal` variant;
panic otherwise.
Fixes#60935.
Rename `LocalInternedString` and more
This PR renames `LocalInternedString` as `SymbolStr`, removes an unnecessary `impl` from it, improves comments, and cleans up some `SymbolStr` uses.
r? @estebank
When a formatting string contains an invalid descriptor, point at it
instead of the argument:
```
error: unknown format trait `foo`
--> $DIR/ifmt-bad-arg.rs:86:17
|
LL | println!("{:foo}", 1);
| ^^^
|
= note: the only appropriate formatting traits are:
- ``, which uses the `Display` trait
- `?`, which uses the `Debug` trait
- `e`, which uses the `LowerExp` trait
- `E`, which uses the `UpperExp` trait
- `o`, which uses the `Octal` trait
- `p`, which uses the `Pointer` trait
- `b`, which uses the `Binary` trait
- `x`, which uses the `LowerHex` trait
- `X`, which uses the `UpperHex` trait
```
(Or more precisely, a pair of such traits: one for `derive(PartialEq)` and one
for `derive(Eq)`.)
((The addition of the second marker trait, `StructuralEq`, is largely a hack to
work-around `fn (&T)` not implementing `PartialEq` and `Eq`; see also issue
rust-lang/rust#46989; otherwise I would just check if `Eq` is implemented.))
Note: this does not use trait fulfillment error-reporting machinery; it just
uses the trait system to determine if the ADT was tagged or not. (Nonetheless, I
have kept an `on_unimplemented` message on the new trait for structural_match
check, even though it is currently not used.)
Note also: this does *not* resolve the ICE from rust-lang/rust#65466, as noted
in a comment added in this commit. Further work is necessary to resolve that and
other problems with the structural match checking, especially to do so without
breaking stable code (adapted from test fn-ptr-is-structurally-matchable.rs):
```rust
fn r_sm_to(_: &SM) {}
fn main() {
const CFN6: Wrap<fn(&SM)> = Wrap(r_sm_to);
let input: Wrap<fn(&SM)> = Wrap(r_sm_to);
match Wrap(input) {
Wrap(CFN6) => {}
Wrap(_) => {}
};
}
```
where we would hit a problem with the strategy of unconditionally checking for
`PartialEq` because the type `for <'a> fn(&'a SM)` does not currently even
*implement* `PartialEq`.
----
added review feedback:
* use an or-pattern
* eschew `return` when tail position will do.
* don't need fresh_expansion; just add `structural_match` to appropriate `allow_internal_unstable` attributes.
also fixed example in doc comment so that it actually compiles.
Add secondary span labels with no text to make it clear when there's a
mismatch bewteen the positional arguments in a format string and the
arguments to the macro. This shouldn't affect experienced users, but it
should make it easier for newcomers to more clearly understand how
`format!()` and `println!()` are supposed to be used.
```
error: 2 positional arguments in format string, but there is 1 argument
--> file8.rs:2:14
|
2 | format!("{} {}", 1);
| ^^ ^^ -
```
instead of
```
error: 2 positional arguments in format string, but there is 1 argument
--> file8.rs:2:14
|
2 | format!("{} {}", 1);
| ^^ ^^
```
Remove implicit dependencies on syntax::pprust
Part of https://github.com/rust-lang/rust/pull/65324.
The main goal here is to facilitate the eventual move of pprust out from libsyntax and because an AST definition typically should not depend on its pretty printer.
r? @estebank
It means an allocation is required to create an empty `TokenStream`, but
all other operations are simpler and marginally faster due to not having
to check for `None`. Overall it simplifies the code for a negligible
performance effect.
The commit also removes `TokenStream::empty` by implementing `Default`,
which is now possible.
Instead just use `pprust::path_to_string(..)` where needed.
This has two benefits:
a) The AST definition is now independent of printing it.
(Therefore we get closer to extracting a data-crate.)
b) Debugging should be easier as program flow is clearer.
Enhance report-time option
## Short overview
This PR is a follow-up to a previously closed#64714 PR.
## Changes introduced by this PR
* `libtest` now retrieves the type of the test within `TestDesc` (available types are: `UnitTest`, `IntegrationTest`, `DocTest`, `Unknown`).
* `--report-time` subcommand of the `libtest` now supports colored output (disabled by default).
* Colorized output depends on the threshold values. Default values (proposed by @wesleywiser):
- For unit-tests: 50ms warn/100ms critical,
- For integration-tests: 500ms warn/1000ms critical,
- For doctests: same as for integration tests,
- For unknown tests: `TEST_WARN_TIMEOUT_S` warn/ `TEST_WARN_TIMEOUT_S * 2` critical (it will only applied single-threaded mode, because otherwise test will be interrupted after reaching `TEST_WARN_TIMEOUT_S`).
- These values can be overrided by setting environment variables (since those thresholds are somewhat constant for every project, it's more flexible to use environment variables than command line arguments).
* New optional flag `--ensure-test-time` for `libtest`. With this flag applied, exectuion time limit excesss will cause test failure.
## What have not been done
There was a comment that it would be nice to have an entry in the Cargo book about it.
However, changes introduced by this PR (and #64663 in which `report-time` flag was added) aren't related directly to `cargo`, it's more about `libtest` itself.
I'm considering that [The Unstable Book](https://doc.rust-lang.org/unstable-book/) is more appropriate place, but not sure if I'm right (and if so, how exactly it should be described).
As one possible option, this PR may be merged without denoting it in the documentation, and in the next PR adding support of this feature to the `cargo` itself, I'll add a note in the Cargo book.
## Scope of this PR
Logical scope of this PR is `libtest` only. However, to get test types, I had to modify also `libsyntax_ext` and `librustdoc` for them to provide information about test type.
## Rationale
Rationale for colored output was submitted in #64714
Providing the information about kind of test was also proposed in #64714, and as an additional benefit this information may be useful for the tools using `libtest` (e.g. `cargo`).
Adding flag to treat time limits excess seems logical to me, so projects that do care about test execution time won't have to invent a wheel.
## Backward compatibility
All the changes are completely backward compatible.
## Demo
r? @wesleywiser
deriving: avoid dummy Span on an artificial `type_ident` path
The dummy Span pointed to the beginning of the source file instead to where the `#[derive]` is located. Later, it tripped the `in_derive_expansion(span)` check at `src/librustc/middle/stability.rs`, causing a span-less deprecation warning to be emitted.
Fixes#56195, Fixes#55417.
The dummy Span pointed to the beginning of the source file instead to
where the `#[derive]` is located. Later, it tripped the
`in_derive_expansion(span)` check at `src/librustc/middle/stability.rs`,
causing a span-less deprecation warning to be emitted.
Fixes#56195, #55417.
