Embed MSVC .natvis files into .pdbs and mangle debuginfo for &str, *T, and [T].
No idea if these changes are reasonable - please feel free to suggest changes/rewrites. And these are some of my first real commits to any rust codebase - *don't* be gentle, and nitpick away, I need to learn! ;)
### Overview
Embedding `.natvis` files into `.pdb`s allows MSVC (and potentially other debuggers) to automatically pick up the visualizers without having to do any additional configuration (other than to perhaps add the relevant .pdb paths to symbol search paths.)
The native debug engine for MSVC parses the type names, making various C++ish assumptions about what they mean and adding various limitations to valid type names. `&str` cannot be matched against a visualizer, but if we emit `str&` instead, it'll be recognized as a reference to a `str`, solving the problem. `[T]` is similarly problematic, but emitting `slice<T>` instead works fine as it looks like a template. I've been unable to get e.g. `slice<u32>&` to match visualizers in VS2015u3, so I've gone with `str*` and `slice<u32>*` instead.
### Possible Issues
* I'm not sure if `slice<T>` is a great mangling for `[T]` or if I should worry about name collisions.
* I'm not sure if `linker.rs` is the right place to be enumerating natvis files.
* I'm not sure if these type name mangling changes should actually be MSVC specific. I recall seeing gdb visualizer tests that might be broken if made more general? I'm hesitant to mess with them without a gdb install. But perhaps I'm just wracking up technical debt.
Should I try `pacman -S mingw-w64-x86_64-gdb` and to make things consistent?
* I haven't touched `const` / `mut` yet, and I'm worried MSVC might trip up on `mut` or their placement.
* I may like terse oneliners too much.
* I don't know if there's broader implications for messing with debug type names here.
* I may have been mistaken about bellow test failures being ignorable / unrelated to this changelist.
### Test Failures on `x86_64-pc-windows-gnu`
```
---- [debuginfo-gdb] debuginfo-gdb\associated-types.rs stdout ----
thread '[debuginfo-gdb] debuginfo-gdb\associated-types.rs' panicked at 'gdb not available but debuginfo gdb debuginfo test requested', src\tools\compiletest\src\runtest.rs:48:16
note: Run with `RUST_BACKTRACE=1` for a backtrace.
[...identical panic causes omitted...]
---- [debuginfo-gdb] debuginfo-gdb\vec.rs stdout ----
thread '[debuginfo-gdb] debuginfo-gdb\vec.rs' panicked at 'gdb not available but debuginfo gdb debuginfo test requested', src\tools\compiletest\src\runtest.rs:48:16
```
### Relevant Issues
* https://github.com/rust-lang/rust/issues/40460 Metaissue for Visual Studio debugging Rust
* https://github.com/rust-lang/rust/issues/36503 Investigate natvis for improved msvc debugging
* https://github.com/PistonDevelopers/VisualRust/issues/160 Debug visualization of Rust data structures
### Pretty Pictures
Fix GDB pretty-printer for tuples and pointers
Names of children should not be the same, because GDB uses them to distinguish the children.
|Before|After|
|---|---|
|||
`main.rs`
```rust
enum Test {
Zero,
One(i32),
Two(i32, String),
Three(i32, String, Vec<String>),
}
fn main() {
let tuple = (1, 2, "Asdfgh");
let zero = Test::Zero;
let one = Test::One(10);
let two = Test::Two(42, "Qwerty".to_owned());
let three = Test::Three(9000,
"Zxcvbn".to_owned(),
vec!["lorem".to_owned(), "ipsum".to_owned(), "dolor".to_owned()]);
println!(""); // breakpoint here
}
```
`launch.json`
```json
{
"version": "0.2.0",
"configurations": [
{
"type": "gdb",
"request": "launch",
"gdbpath": "rust-gdb",
"name": "Launch Program",
"valuesFormatting": "prettyPrinters", //this requires plugin Native Debug >= 0.20.0
"target": "./target/debug/test_pretty_printers",
"cwd": "${workspaceRoot}"
}
]
}
```
Use a class without children() method for printing empty structs.
Presence of this method makes GDB's variable objects interface act like
if the struct had children.
travis: Parallelize tests on Android
Currently our slowest test suite on android, run-pass, takes over 5 times longer
than the x86_64 component (~400 -> ~2200s). Typically QEMU emulation does indeed
add overhead, but not 5x for this kind of workload. One of the slowest parts of
the Android process is that *compilation* happens serially. Tests themselves
need to run single-threaded on the emulator (due to how the test harness works)
and this forces the compiles themselves to be single threaded.
Now Travis gives us more than one core per machine, so it'd be much better if we
could take advantage of them! The emulator itself is still fundamentally
single-threaded, but we should see a nice speedup by sending binaries for it to
run much more quickly.
It turns out that we've already got all the toos to do this in-tree. The
qemu-test-{server,client} that are in use for the ARM Linux testing are a
perfect match for the Android emulator. This commit migrates the custom adb
management code in compiletest/rustbuild to the same qemu-test-{server,client}
implementation that ARM Linux uses.
This allows us to lift the parallelism restriction on the compiletest test
suites, namely run-pass. Consequently although we'll still basically run the
tests themselves in single threaded mode we'll be able to compile all of them in
parallel, keeping the pipeline much more full hopefully and using more cores for
the work at hand. Additionally the architecture here should be a bit speedier as
it should have less overhead than adb which is a whole new process on both the
host and the emulator!
Locally on an 8 core machine I've seen the run-pass test suite speed up from
taking nearly an hour to only taking 5 minutes. I don't think we'll see quite a
drastic speedup on Travis but I'm hoping this change can place the Android tests
well below 2 hours instead of just above 2 hours.
Because the client/server here are now repurposed for more than just QEMU,
they've been renamed to `remote-test-{server,client}`.
Note that this PR does not currently modify how debuginfo tests are executed on
Android. While parallelizable it wouldn't be quite as easy, so that's left to
another day. Thankfull that test suite is much smaller than the run-pass test
suite.
Currently our slowest test suite on android, run-pass, takes over 5 times longer
than the x86_64 component (~400 -> ~2200s). Typically QEMU emulation does indeed
add overhead, but not 5x for this kind of workload. One of the slowest parts of
the Android process is that *compilation* happens serially. Tests themselves
need to run single-threaded on the emulator (due to how the test harness works)
and this forces the compiles themselves to be single threaded.
Now Travis gives us more than one core per machine, so it'd be much better if we
could take advantage of them! The emulator itself is still fundamentally
single-threaded, but we should see a nice speedup by sending binaries for it to
run much more quickly.
It turns out that we've already got all the tools to do this in-tree. The
qemu-test-{server,client} that are in use for the ARM Linux testing are a
perfect match for the Android emulator. This commit migrates the custom adb
management code in compiletest/rustbuild to the same qemu-test-{server,client}
implementation that ARM Linux uses.
This allows us to lift the parallelism restriction on the compiletest test
suites, namely run-pass. Consequently although we'll still basically run the
tests themselves in single threaded mode we'll be able to compile all of them in
parallel, keeping the pipeline much more full and using more cores for the work
at hand. Additionally the architecture here should be a bit speedier as it
should have less overhead than adb which is a whole new process on both the host
and the emulator!
Locally on an 8 core machine I've seen the run-pass test suite speed up from
taking nearly an hour to only taking 6 minutes. I don't think we'll see quite a
drastic speedup on Travis but I'm hoping this change can place the Android tests
well below 2 hours instead of just above 2 hours.
Because the client/server here are now repurposed for more than just QEMU,
they've been renamed to `remote-test-{server,client}`.
Note that this PR does not currently modify how debuginfo tests are executed on
Android. While parallelizable it wouldn't be quite as easy, so that's left to
another day. Thankfully that test suite is much smaller than the run-pass test
suite.
As a final fix I discovered that the ARM and Android test suites were actually
running all library unit tests (e.g. stdtest, coretest, etc) twice. I've
corrected that to only run tests once which should also give a nice boost in
overall cycle time here.
Recently we switched from the win32 MinGW toolchain to the pthreads-based
toolchain. We ship `gcc.exe` from this toolchain with the `rust-mingw` package
in the standard distribution but the pthreads version of `gcc.exe` depends on
`libwinpthread-1.dll`. While we're shipping this DLL for the compiler to depend
on we're not shipping it for gcc. As a workaround just copy the dll to gcc.exe
location and don't attempt to share for now.
cc https://github.com/rust-lang/rust/issues/31840#issuecomment-297478538
Instead of hard-coding the command to run, using the environment
variable `GDB_CMD` (that defaults to `gdb`) allows using a different
debugger than the default `gdb` executable.
This gives the possibility to use `cgdb` as the debugger, which provides
a nicer user interface. Note that one has to use `GDB_CMD="cgdb --"` to
use cgdb (note the trailing `--`) to let cgdb pass the proper arguments
to `gdb`.
This commit shrinks the size of the aforementioned table from
2,102 bytes to 1,197 bytes. This is achieved by an observation that
most u16 entries are common in its upper byte. Specifically:
- SINGLETONS now uses two tables, one for (upper byte, lower count)
and another for a series of lower bytes. For each upper byte given
number of lower bytes are read and compared.
- NORMAL now uses a variable length format for the count of "true"
codepoints and "false" codepoints (one byte with MSB unset, or
two big-endian bytes with the first MSB set).
The code size and relative performance roughly remains same as this
commit tries to optimize for both. The new table and algorithm has
been verified for the equivalence to older ones.
Add intrinsics & target features for rd{rand,seed}
One question is whether or not we want to map feature name `rdrnd` to `rdrand` instead.
EDIT: as for use case, I would like to port my rdrand crate from inline assembly to these intrinsics.
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
Reduce the size of static data in std_unicode::tables
`BoolTrie` works well for sets of code points spread out through most of Unicode’s range, but is uses a lot of space for sets with few, mostly low, code points.
This switches a few of its instances to a similar but simpler trie data structure.
CC @raphlinus, who wrote the original `BoolTrie`.
## Before
`size_of::<BoolTrie>()` is 1552, which is added to `table.r3.len() * 8 + t.r5.len() + t.r6.len() * 8`:
* `Cc_table`: 1632
* `White_Space_table`: 1656
* `Pattern_White_Space_table`: 1640
* Total: 4928 bytes
## After
`size_of::<SmallBoolTrie>()` is 32, which is added to `t.r1.len() + t.r2.len() * 8`:
* `Cc_table`: 51
* `White_Space_table`: 273
* `Pattern_White_Space_table`: 193
* Total: 517 bytes
## Difference
Every Rust program with `std` statically linked should be about 4 KB smaller.
`BoolTrie` works well for sets of code points spread out through
most of Unicode’s range, but is uses a lot of space for sets
with few, mostly low, code points.
This switches a few of its instances to a similar but simpler trie
data structure.
## Before
`size_of::<BoolTrie>()` is 1552, which is added to
`table.r3.len() * 8 + t.r5.len() + t.r6.len() * 8`:
* `Cc_table`: 1632
* `White_Space_table`: 1656
* `Pattern_White_Space_table`: 1640
* Total: 4928 bytes
## After
`size_of::<SmallBoolTrie>()` is 32, which is added to
`t.r1.len() + t.r2.len() * 8`:
* `Cc_table`: 51
* `White_Space_table`: 273
* `Pattern_White_Space_table`: 193
* Total: 517 bytes
## Difference
Every Rust program with `std` statically linked should be about 4 KB smaller.
