resolve: Implement prelude search for macro paths, implement tool attributes
When identifier is macro path is resolved in scopes (i.e. the first path segment - `foo` in `foo::mac!()` or `foo!()`), scopes are searched in the same order as for non-macro paths - items in modules, extern prelude, tool prelude (see later), standard library prelude, language prelude, but with some extra shadowing restrictions (names from globs and macro expansions cannot shadow names from outer scopes). See the comment in `fn resolve_lexical_macro_path_segment` for more details.
"Tool prelude" currently contains two "tool modules" `rustfmt` and `clippy`, and is searched immediately after extern prelude.
This makes the [possible long-term solution](https://github.com/rust-lang/rfcs/blob/master/text/2103-tool-attributes.md#long-term-solution) for tool attributes exactly equivalent to the existing extern prelude scheme, except that `--extern=my_crate` making crate names available in scope is replaced with something like `--tool=my_tool` making tool names available in scope.
The `tool_attributes` feature is still unstable and `#![feature(tool_attributes)]` now implicitly enables `#![feature(use_extern_macros)]`. `use_extern_macros` is a prerequisite for `tool_attributes`, so their stabilization will happen in the same order.
If `use_extern_macros` is not enabled, then tool attributes are treated as custom attributes (this is temporary, anyway).
Fixes https://github.com/rust-lang/rust/issues/52576
Fixes https://github.com/rust-lang/rust/issues/52512
Fixes https://github.com/rust-lang/rust/issues/51277
cc https://github.com/rust-lang/rust/issues/52269
Rollup of 30 pull requests
Successful merges:
- #52340 (Document From trait implementations for OsStr, OsString, CString, and CStr)
- #52628 (Cleanup some rustdoc code)
- #52732 (Remove unstable and deprecated APIs)
- #52745 (Update clippy to latest master)
- #52771 (Clarify thread::park semantics)
- #52778 (Improve readability of serialize.rs)
- #52810 ([NLL] Don't make "fake" match variables mutable)
- #52821 (pretty print for std::collections::vecdeque)
- #52822 (Fix From<LocalWaker>)
- #52824 (Fix -Wpessimizing-move warnings in rustllvm/PassWrapper)
- #52825 (Make sure #47772 does not regress)
- #52831 (remove references to AUTHORS.txt file)
- #52842 (update comment)
- #52846 (Add timeout to use of `curl` in bootstrap.py.)
- #52851 (Make the tool_lints actually usable)
- #52853 (Improve bootstrap help on stages)
- #52859 (Use Vec::extend in SmallVec::extend when applicable)
- #52861 (Add targets for HermitCore (https://hermitcore.org) to the Rust compiler and port libstd to it.)
- #52867 (releases.md: fix 2 typos)
- #52870 (Implement Unpin for FutureObj and LocalFutureObj)
- #52876 (run-pass/const-endianness: negate before to_le())
- #52878 (Fix wrong issue number in the test name)
- #52883 (Include lifetime in mutability suggestion in NLL messages)
- #52888 (Use suggestions for shell format arguments)
- #52904 (NLL: sort diagnostics by span)
- #52905 (Fix a typo in unsize.rs)
- #52907 (NLL: On "cannot move out of type" error, print original before rewrite)
- #52914 (Only run the sparc-abi test on sparc)
- #52918 (Backport 1.27.2 release notes)
- #52929 (Update compatibility note for 1.28.0 to be correct)
Failed merges:
r? @ghost
Recent changes to lints disallowed lints from being emitted against code located
in foreign macros, except for future-incompatible lints. For a future
incompatible lint, however, the automatic suggestions may not be applicable!
This commit updates this code path to force all applicability suggestions made
to foreign macros to never be `MachineApplicable`. This should avoid rustfix
actually attempting fixing these suggestions, causing non-compiling code to be
produced.
Closesrust-lang/cargo#5799
rustc: Implement tokenization of nested items
Ever plagued by #43081 the compiler can return surprising spans in situations
related to procedural macros. This is exhibited by #47983 where whenever a
procedural macro is invoked in a nested item context it would fail to have
correct span information.
While #43230 provided a "hack" to cache the token stream used for each item in
the compiler it's not a full-blown solution. This commit continues to extend
this "hack" a bit more to work for nested items.
Previously in the parser the `parse_item` method would collect the tokens for an
item into a cache on the item itself. It turned out, however, that nested items
were parsed through the `parse_item_` method, so they didn't receive similar
treatment. To remedy this situation the hook for collecting tokens was moved
into `parse_item_` instead of `parse_item`.
Afterwards the token collection scheme was updated to support nested collection
of tokens. This is implemented by tracking `TokenStream` tokens instead of
`TokenTree` to allow for collecting items into streams at intermediate layers
and having them interleaved in the upper layers.
All in all, this...
Closes#47983
Ever plagued by #43081 the compiler can return surprising spans in situations
related to procedural macros. This is exhibited by #47983 where whenever a
procedural macro is invoked in a nested item context it would fail to have
correct span information.
While #43230 provided a "hack" to cache the token stream used for each item in
the compiler it's not a full-blown solution. This commit continues to extend
this "hack" a bit more to work for nested items.
Previously in the parser the `parse_item` method would collect the tokens for an
item into a cache on the item itself. It turned out, however, that nested items
were parsed through the `parse_item_` method, so they didn't receive similar
treatment. To remedy this situation the hook for collecting tokens was moved
into `parse_item_` instead of `parse_item`.
Afterwards the token collection scheme was updated to support nested collection
of tokens. This is implemented by tracking `TokenStream` tokens instead of
`TokenTree` to allow for collecting items into streams at intermediate layers
and having them interleaved in the upper layers.
All in all, this...
Closes#47983
This commit updates the tokenization of items which are subsequently passed to
`proc_macro` to ensure that span information is preserved on attributes as much
as possible. Previously this area of the code suffered from #43081 where we
haven't actually implemented converting an attribute to to a token tree yet, but
a local fix was possible here.
Closes#47941
This commit stabilizes some of the `proc_macro` language feature as well as a
number of APIs in the `proc_macro` crate as [previously discussed][1]. This
means that on stable Rust you can now define custom procedural macros which
operate as attributes attached to items or `macro_rules!`-like bang-style
invocations. This extends the suite of currently stable procedural macros,
custom derives, with custom attributes and custom bang macros.
Note though that despite the stabilization in this commit procedural macros are
still not usable on stable Rust. To stabilize that we'll need to stabilize at
least part of the `use_extern_macros` feature. Currently you can define a
procedural macro attribute but you can't import it to call it!
A summary of the changes made in this PR (as well as the various consequences)
is:
* The `proc_macro` language and library features are now stable.
* Other APIs not stabilized in the `proc_macro` crate are now named under a
different feature, such as `proc_macro_diagnostic` or `proc_macro_span`.
* A few checks in resolution for `proc_macro` being enabled have switched over
to `use_extern_macros` being enabled. This means that code using
`#![feature(proc_macro)]` today will likely need to move to
`#![feature(use_extern_macros)]`.
It's intended that this PR, once landed, will be followed up with an attempt to
stabilize a small slice of `use_extern_macros` just for procedural macros to
make this feature 100% usable on stable.
[1]: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252
resolve: Functions introducing procedural macros reserve a slot in the macro namespace as well
Similarly to https://github.com/rust-lang/rust/pull/52234, this gives us symmetry between internal and external views of a crate, but in this case it's always an error to call a procedural macro in the same crate in which it's defined.
Closes https://github.com/rust-lang/rust/issues/52225
rustc: Tweak expansion of #[proc_macro] for 2018
The syntactical expansion of `#[proc_macro]` and related attributes currently
contains absolute paths which conflicts with a lint for the 2018 edition,
causing issues like #52214. This commit puts a band-aid on the issue by ensuring
that procedural macros can also migrate to the 2018 edition for now by tweaking
the expansion based on what features are activated. A more long-term solution
would probably tweak the edition hygiene of spans, but this should do the trick
for now.
Closes#52214
The syntactical expansion of `#[proc_macro]` and related attributes currently
contains absolute paths which conflicts with a lint for the 2018 edition,
causing issues like #52214. This commit puts a band-aid on the issue by ensuring
that procedural macros can also migrate to the 2018 edition for now by tweaking
the expansion based on what features are activated. A more long-term solution
would probably tweak the edition hygiene of spans, but this should do the trick
for now.
Closes#52214
rustc: Verify #[proc_macro] is only a word
... and perform the same verification for #[proc_macro_attribute], currently
neither of these attributes take any arguments.
Closes#52273
This commit updates the `Mac_` AST structure to keep track of the delimiters
that it originally had for its invocation. This allows us to faithfully
pretty-print macro invocations not using parentheses (e.g. `vec![...]`). This in
turn helps procedural macros due to #43081.
Closes#50840
- The output of each test is now in its own directory.
- "auxiliary" output is now under the respective test directory.
- `stage_id` removed from filenames, and instead placed in the stamp file as a hash. This helps keep path lengths down for Windows.
In brief, the new layout looks like this:
```
<build_base>/<relative_dir>/<testname>.<revision>.<mode>/
stamp
<testname>.err
<testname>.out
a (binary)
auxiliary/lib<auxname>.dylib
auxiliary/<auxname>/<auxname>.err
auxiliary/<auxname>/<auxname>.out
```
(revision and mode are optional)
rustc: Include semicolon when removing `extern crate`
Currently the lint for removing `extern crate` suggests removing `extern crate`
most of the time, but the rest of the time it suggest replacing it with `use
crate_name`. Unfortunately though when spliced into the original code you're
replacing
extern crate foo;
with
use foo
which is syntactically invalid! This commit ensure that the trailing semicolon
is included in rustc's suggestion to ensure that the code continues to compile
afterwards.
This commit updates one of the edition lints to only suggest deleting `extern
crate` if it actually works. Otherwise this can yield some confusing behavior
with rustfix specifically where if you accidentally deny the `rust_2018_idioms`
lint in the 2015 edition it's suggesting features that don't work!
Currently the lint for removing `extern crate` suggests removing `extern crate`
most of the time, but the rest of the time it suggest replacing it with `use
crate_name`. Unfortunately though when spliced into the original code you're
replacing
extern crate foo;
with
use foo
which is syntactically invalid! This commit ensure that the trailing semicolon
is included in rustc's suggestion to ensure that the code continues to compile
afterwards.
This commit starts to lay some groundwork for the stabilization of custom
attribute invocations and general procedural macros. It applies a number of
changes discussed on [internals] as well as a [recent issue][issue], namely:
* The path used to specify a custom attribute must be of length one and cannot
be a global path. This'll help future-proof us against any ambiguities and
give us more time to settle the precise syntax. In the meantime though a bare
identifier can be used and imported to invoke a custom attribute macro. A new
feature gate, `proc_macro_path_invoc`, was added to gate multi-segment paths
and absolute paths.
* The set of items which can be annotated by a custom procedural attribute has
been restricted. Statements, expressions, and modules are disallowed behind
two new feature gates: `proc_macro_expr` and `proc_macro_mod`.
* The input to procedural macro attributes has been restricted and adjusted.
Today an invocation like `#[foo(bar)]` will receive `(bar)` as the input token
stream, but after this PR it will only receive `bar` (the delimiters were
removed). Invocations like `#[foo]` are still allowed and will be invoked in
the same way as `#[foo()]`. This is a **breaking change** for all nightly
users as the syntax coming in to procedural macros will be tweaked slightly.
* Procedural macros (`foo!()` style) can only be expanded to item-like items by
default. A separate feature gate, `proc_macro_non_items`, is required to
expand to items like expressions, statements, etc.
Closes#50038
[internals]: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252
[issue]: https://github.com/rust-lang/rust/issues/50038
This commit is a reorganization of the `proc_macro` crate's public user-facing
API. This is the result of a number of discussions at the recent Rust All-Hands
where we're hoping to get the `proc_macro` crate into ship shape for
stabilization of a subset of its functionality in the Rust 2018 release.
The reorganization here is motivated by experiences from the `proc-macro2`,
`quote`, and `syn` crates on crates.io (and other crates which depend on them).
The main focus is future flexibility along with making a few more operations
consistent and/or fixing bugs. A summary of the changes made from today's
`proc_macro` API is:
* The `TokenNode` enum has been removed and the public fields of `TokenTree`
have also been removed. Instead the `TokenTree` type is now a public enum
(what `TokenNode` was) and each variant is an opaque struct which internally
contains `Span` information. This makes the various tokens a bit more
consistent, require fewer wrappers, and otherwise provides good
future-compatibility as opaque structs are easy to modify later on.
* `Literal` integer constructors have been expanded to be unambiguous as to what
they're doing and also allow for more future flexibility. Previously
constructors like `Literal::float` and `Literal::integer` were used to create
unsuffixed literals and the concrete methods like `Literal::i32` would create
a suffixed token. This wasn't immediately clear to all users (the
suffixed/unsuffixed aspect) and having *one* constructor for unsuffixed
literals required us to pick a largest type which may not always be true. To
fix these issues all constructors are now of the form
`Literal::i32_unsuffixed` or `Literal::i32_suffixed` (for all integral types).
This should allow future compatibility as well as being immediately clear
what's suffixed and what isn't.
* Each variant of `TokenTree` internally contains a `Span` which can also be
configured via `set_span`. For example `Literal` and `Term` now both
internally contain a `Span` rather than having it stored in an auxiliary
location.
* Constructors of all tokens are called `new` now (aka `Term::intern` is gone)
and most do not take spans. Manufactured tokens typically don't have a fresh
span to go with them and the span is purely used for error-reporting
**except** the span for `Term`, which currently affects hygiene. The default
spans for all these constructed tokens is `Span::call_site()` for now.
The `Term` type's constructor explicitly requires passing in a `Span` to
provide future-proofing against possible hygiene changes. It's intended that a
first pass of stabilization will likely only stabilize `Span::call_site()`
which is an explicit opt-in for "I would like no hygiene here please". The
intention here is to make this explicit in procedural macros to be
forwards-compatible with a hygiene-specifying solution.
* Some of the conversions for `TokenStream` have been simplified a little.
* The `TokenTreeIter` iterator was renamed to `token_stream::IntoIter`.
Overall the hope is that this is the "final pass" at the API of `TokenStream`
and most of `TokenTree` before stabilization. Explicitly left out here is any
changes to `Span`'s API which will likely need to be re-evaluated before
stabilization.
All changes in this PR have already been reflected to the [`proc-macro2`],
`quote`, and `syn` crates. New versions of all these crates have also been
published to crates.io.
Once this lands in nightly I plan on making an internals post again summarizing
the changes made here and also calling on all macro authors to give the APIs a
spin and see how they work. Hopefully pending no major issues we can then have
an FCP to stabilize later this cycle!
[`proc-macro2`]: https://docs.rs/proc-macro2/0.3.1/proc_macro2/
