Auto merge of #52618 - alexcrichton:capture-more, r=petrochenkov

rustc: Implement tokenization of nested items

Ever plagued by #43081 the compiler can return surprising spans in situations
related to procedural macros. This is exhibited by #47983 where whenever a
procedural macro is invoked in a nested item context it would fail to have
correct span information.

While #43230 provided a "hack" to cache the token stream used for each item in
the compiler it's not a full-blown solution. This commit continues to extend
this "hack" a bit more to work for nested items.

Previously in the parser the `parse_item` method would collect the tokens for an
item into a cache on the item itself. It turned out, however, that nested items
were parsed through the `parse_item_` method, so they didn't receive similar
treatment. To remedy this situation the hook for collecting tokens was moved
into `parse_item_` instead of `parse_item`.

Afterwards the token collection scheme was updated to support nested collection
of tokens. This is implemented by tracking `TokenStream` tokens instead of
`TokenTree` to allow for collecting items into streams at intermediate layers
and having them interleaved in the upper layers.

All in all, this...

Closes #47983

src/test/ui-fulldeps/proc-macro/auxiliary/nested-item-spans.rs

@@ -0,0 +1,22 @@
+// Copyright 2018 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// no-prefer-dynamic
+
+#![crate_type = "proc-macro"]
+
+extern crate proc_macro;
+
+use proc_macro::*;
+
+#[proc_macro_attribute]
+pub fn foo(_: TokenStream, item: TokenStream) -> TokenStream {
+    item.into_iter().collect()
+}

src/test/ui-fulldeps/proc-macro/nested-item-spans.rs

@@ -0,0 +1,36 @@
+// Copyright 2018 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// aux-build:nested-item-spans.rs
+
+#![feature(use_extern_macros)]
+
+extern crate nested_item_spans;
+
+use nested_item_spans::foo;
+
+#[foo]
+fn another() {
+    fn bar() {
+        let x: u32 = "x"; //~ ERROR: mismatched types
+    }
+
+    bar();
+}
+
+fn main() {
+    #[foo]
+    fn bar() {
+        let x: u32 = "x"; //~ ERROR: mismatched types
+    }
+
+    bar();
+    another();
+}

src/test/ui-fulldeps/proc-macro/nested-item-spans.stderr

@@ -0,0 +1,21 @@
+error[E0308]: mismatched types
+  --> $DIR/nested-item-spans.rs:22:22
+   |
+LL |         let x: u32 = "x"; //~ ERROR: mismatched types
+   |                      ^^^ expected u32, found reference
+   |
+   = note: expected type `u32`
+              found type `&'static str`
+
+error[E0308]: mismatched types
+  --> $DIR/nested-item-spans.rs:31:22
+   |
+LL |         let x: u32 = "x"; //~ ERROR: mismatched types
+   |                      ^^^ expected u32, found reference
+   |
+   = note: expected type `u32`
+              found type `&'static str`
+
+error: aborting due to 2 previous errors
+
+For more information about this error, try `rustc --explain E0308`.