resolve: Implement prelude search for macro paths, implement tool attributes
When identifier is macro path is resolved in scopes (i.e. the first path segment - `foo` in `foo::mac!()` or `foo!()`), scopes are searched in the same order as for non-macro paths - items in modules, extern prelude, tool prelude (see later), standard library prelude, language prelude, but with some extra shadowing restrictions (names from globs and macro expansions cannot shadow names from outer scopes). See the comment in `fn resolve_lexical_macro_path_segment` for more details.
"Tool prelude" currently contains two "tool modules" `rustfmt` and `clippy`, and is searched immediately after extern prelude.
This makes the [possible long-term solution](https://github.com/rust-lang/rfcs/blob/master/text/2103-tool-attributes.md#long-term-solution) for tool attributes exactly equivalent to the existing extern prelude scheme, except that `--extern=my_crate` making crate names available in scope is replaced with something like `--tool=my_tool` making tool names available in scope.
The `tool_attributes` feature is still unstable and `#![feature(tool_attributes)]` now implicitly enables `#![feature(use_extern_macros)]`. `use_extern_macros` is a prerequisite for `tool_attributes`, so their stabilization will happen in the same order.
If `use_extern_macros` is not enabled, then tool attributes are treated as custom attributes (this is temporary, anyway).
Fixes https://github.com/rust-lang/rust/issues/52576
Fixes https://github.com/rust-lang/rust/issues/52512
Fixes https://github.com/rust-lang/rust/issues/51277
cc https://github.com/rust-lang/rust/issues/52269
Reexport tests without polluting namespaces
This should fix issue #52557.
Basically now we gensym a new name for the test function and reexport that.
That way the test function's reexport name can't conflict because it was impossible for the test author to write it down.
We then use a `use` statement to expose the original name using the original visibility.
Tweak the raw_identifiers lints in 2018
* Enable the `raw_identifiers` feature automatically in the 2018 preview
* Only emit lint warnings if the `raw_identifiers` feature is activated
cc rust-lang/cargo#5783
Replace push loops with extend() where possible
Or set the vector capacity where I couldn't do it.
According to my [simple benchmark](https://gist.github.com/ljedrz/568e97621b749849684c1da71c27dceb) `extend`ing a vector can be over **10 times** faster than `push`ing to it in a loop:
10 elements (6.1 times faster):
```
test bench_extension ... bench: 75 ns/iter (+/- 23)
test bench_push_loop ... bench: 458 ns/iter (+/- 142)
```
100 elements (11.12 times faster):
```
test bench_extension ... bench: 87 ns/iter (+/- 26)
test bench_push_loop ... bench: 968 ns/iter (+/- 3,528)
```
1000 elements (11.04 times faster):
```
test bench_extension ... bench: 311 ns/iter (+/- 9)
test bench_push_loop ... bench: 3,436 ns/iter (+/- 233)
```
Seems like a good idea to use `extend` as much as possible.
Add the -Zcrate-attr=foo unstable rustc option
This PR adds a new unstable option to `rustc`: `-Zcrate-attr=foo`. The option can be used to inject crate-level attributes from the CLI, and it's meant to be used by tools like Crater that needs to add their own attributes to a crate without changing the source code.
The exact reason I need this is to implement "edition runs" in Crater: we need to add the preview feature flag to every crate, and editing the crates' source code on the fly might produce unexpected results, while a compiler flag is more reliable.
cc https://github.com/rust-lang-nursery/crater/issues/282 @Mark-Simulacrum
Rollup of bare_trait_objects PRs
All deny attributes were moved into bootstrap so they can be disabled with a line of config.
Warnings for external tools are allowed and it's up to the tool's maintainer to keep it warnings free.
r? @Mark-Simulacrum
cc @ljedrz @kennytm
* Enable the `raw_identifiers` feature automatically in the 2018 preview
* Only emit lint warnings if the `raw_identifiers` feature is activated
cc rust-lang/cargo#5783
rustc: Implement tokenization of nested items
Ever plagued by #43081 the compiler can return surprising spans in situations
related to procedural macros. This is exhibited by #47983 where whenever a
procedural macro is invoked in a nested item context it would fail to have
correct span information.
While #43230 provided a "hack" to cache the token stream used for each item in
the compiler it's not a full-blown solution. This commit continues to extend
this "hack" a bit more to work for nested items.
Previously in the parser the `parse_item` method would collect the tokens for an
item into a cache on the item itself. It turned out, however, that nested items
were parsed through the `parse_item_` method, so they didn't receive similar
treatment. To remedy this situation the hook for collecting tokens was moved
into `parse_item_` instead of `parse_item`.
Afterwards the token collection scheme was updated to support nested collection
of tokens. This is implemented by tracking `TokenStream` tokens instead of
`TokenTree` to allow for collecting items into streams at intermediate layers
and having them interleaved in the upper layers.
All in all, this...
Closes#47983
Ever plagued by #43081 the compiler can return surprising spans in situations
related to procedural macros. This is exhibited by #47983 where whenever a
procedural macro is invoked in a nested item context it would fail to have
correct span information.
While #43230 provided a "hack" to cache the token stream used for each item in
the compiler it's not a full-blown solution. This commit continues to extend
this "hack" a bit more to work for nested items.
Previously in the parser the `parse_item` method would collect the tokens for an
item into a cache on the item itself. It turned out, however, that nested items
were parsed through the `parse_item_` method, so they didn't receive similar
treatment. To remedy this situation the hook for collecting tokens was moved
into `parse_item_` instead of `parse_item`.
Afterwards the token collection scheme was updated to support nested collection
of tokens. This is implemented by tracking `TokenStream` tokens instead of
`TokenTree` to allow for collecting items into streams at intermediate layers
and having them interleaved in the upper layers.
All in all, this...
Closes#47983
Improve suggestion for missing fmt str in println
Avoid using `concat!(fmt, "\n")` to improve the diagnostics being
emitted when the first `println!()` argument isn't a formatting string
literal.
Fix#52347.
Prepare proc_macro for decoupling it from the rest of the compiler.
This is #49219 up to the point where the bridge is introduced. Aside from moving some code around, the largest change is the rewrite of `proc_macro::quote` to be simpler and do less introspection.
I'd like to also extend `quote!` with `${stmt;...;expr}` instead of just `$variable` (and maybe even `$(... $iter ...)*`), which seems pretty straight-forward now, but I don't know if/when I should.
r? @alexcrichton or @dtolnay cc @jseyfried @petrochenkov
proc_macro: Preserve spans of attributes on functions
This commit updates the tokenization of items which are subsequently passed to
`proc_macro` to ensure that span information is preserved on attributes as much
as possible. Previously this area of the code suffered from #43081 where we
haven't actually implemented converting an attribute to to a token tree yet, but
a local fix was possible here.
Closes#47941
