This avoids double compiled Cargo. Hopefully this would speed up (extended) compilation for ~10m.
Notes: when updating Cargo submodule, the replacement version may also need to be updated.
Update to the Rusty rust-installer
This updates the rust-installer submodule to the new version written in Rust (rust-lang/rust-installer#62), now moved to `src/tools/rust-installer` and invoked in `dist.rs` as a cargo-based tool command. All of the former shell-script invocations now invoke the tool, otherwise keeping the same arguments as before.
As a small bonus, `rustc-src` now also uses the same tarball generator, so it gains a smaller `.tar.xz` too.
Fixes#41569. r? @alexcrichton
linkchecker: Add support for <base> tag
Add support for the HTML <base> tag as used by mdBook so The Unstable
Book can be checked.
Also cleanup a few things:
* Stop checking the name attribute. It should never have been used and
mdBook has since been fixed not to use it.
* Make sure we only check html files.
* Remove a few unnecessary allocations.
Finally, dead links in The Unstable Book have been fixed.
Add support for the HTML <base> tag as used by mdBook so The Unstable
Book can be checked.
Also cleanup a few things:
* Stop checking the name attribute. It should never have been used and
mdBook has since been fixed not to use it.
* Make sure we only check html files.
* Remove a few unnecessary allocations.
Finally, dead links in The Unstable Book have been fixed.
Update RLS to solve serious crash
As a follow-up to rust-lang-nursery/rls#312, this PR contains the fix to the latest experienced RLS crashes, which should hopefully reach the nightly release as soon as possible.
ci: Update android ndk and sdk
Make install-sdk.sh and install-ndk.sh more generic so future updates can be made directly on Dockerfile. Update ndk to r13b, which will be necessary to make host builds for android (in the future). Update sdk to r25.2.5 (maybe some emulator performance improvement).
Generate XZ-compressed tarballs
Integrate the new `rust-installer` and extend manifests with keys for xz-compressed tarballs.
One of the steps required for https://github.com/rust-lang/rust/issues/21724
Overall goal: reduce the amount of context a mir pass needs so that it
resembles a query.
- The hooks are no longer "threaded down" to the pass, but rather run
automatically from the top-level (we also thread down the current pass
number, so that the files are sorted better).
- The hook now receives a *single* callback, rather than a callback per-MIR.
- The traits are no longer lifetime parameters, which moved to the
methods -- given that we required
`for<'tcx>` objecs, there wasn't much point to that.
- Several passes now store a `String` instead of a `&'l str` (again, no
point).
Update stage0 bootstrap compiler
We've got a freshly minted beta compiler, let's update to use that on nightly!
This has a few other changes associated with it as well
* A bump to the rustc version number (to 1.19.0)
* Movement of the `cargo` and `rls` submodules to their "proper" location in
`src/tools/{cargo,rls}`. Now that Cargo workspaces support the `exclude`
option this can work.
* Updates of the `cargo` and `rls` submodules to their master branches.
* Tweak to the `src/stage0.txt` format to be more amenable for Cargo version
numbers. On the beta channel Cargo will bootstrap from a different version
than rustc (e.g. the version numbers are different), so we need different
configuration for this.
* Addition of `dev` as a readable key in the `src/stage0.txt` format. If present
then stage0 compilers are downloaded from `dev-static.rust-lang.org` instead
of `static.rust-lang.org`. This is added to accomodate our updated release
process with Travis and AppVeyor.
We've got a freshly minted beta compiler, let's update to use that on nightly!
This has a few other changes associated with it as well
* A bump to the rustc version number (to 1.19.0)
* Movement of the `cargo` and `rls` submodules to their "proper" location in
`src/tools/{cargo,rls}`. Now that Cargo workspaces support the `exclude`
option this can work.
* Updates of the `cargo` and `rls` submodules to their master branches.
* Tweak to the `src/stage0.txt` format to be more amenable for Cargo version
numbers. On the beta channel Cargo will bootstrap from a different version
than rustc (e.g. the version numbers are different), so we need different
configuration for this.
* Addition of `dev` as a readable key in the `src/stage0.txt` format. If present
then stage0 compilers are downloaded from `dev-static.rust-lang.org` instead
of `static.rust-lang.org`. This is added to accomodate our updated release
process with Travis and AppVeyor.
use diff crate for compile-fail test diagnostics #41474
Hello!
This fixes#41474
We were using a custom implementation to dump the differences between expected and actual outputs of compile-fail tests.
I removed this internal implementation and added `diff` crate as a new dependency to `compile-fail`.
Again, huge thanks to @nikomatsakis for guiding.
travis: Parallelize tests on Android
Currently our slowest test suite on android, run-pass, takes over 5 times longer
than the x86_64 component (~400 -> ~2200s). Typically QEMU emulation does indeed
add overhead, but not 5x for this kind of workload. One of the slowest parts of
the Android process is that *compilation* happens serially. Tests themselves
need to run single-threaded on the emulator (due to how the test harness works)
and this forces the compiles themselves to be single threaded.
Now Travis gives us more than one core per machine, so it'd be much better if we
could take advantage of them! The emulator itself is still fundamentally
single-threaded, but we should see a nice speedup by sending binaries for it to
run much more quickly.
It turns out that we've already got all the toos to do this in-tree. The
qemu-test-{server,client} that are in use for the ARM Linux testing are a
perfect match for the Android emulator. This commit migrates the custom adb
management code in compiletest/rustbuild to the same qemu-test-{server,client}
implementation that ARM Linux uses.
This allows us to lift the parallelism restriction on the compiletest test
suites, namely run-pass. Consequently although we'll still basically run the
tests themselves in single threaded mode we'll be able to compile all of them in
parallel, keeping the pipeline much more full hopefully and using more cores for
the work at hand. Additionally the architecture here should be a bit speedier as
it should have less overhead than adb which is a whole new process on both the
host and the emulator!
Locally on an 8 core machine I've seen the run-pass test suite speed up from
taking nearly an hour to only taking 5 minutes. I don't think we'll see quite a
drastic speedup on Travis but I'm hoping this change can place the Android tests
well below 2 hours instead of just above 2 hours.
Because the client/server here are now repurposed for more than just QEMU,
they've been renamed to `remote-test-{server,client}`.
Note that this PR does not currently modify how debuginfo tests are executed on
Android. While parallelizable it wouldn't be quite as easy, so that's left to
another day. Thankfull that test suite is much smaller than the run-pass test
suite.
Implement a file-path remapping feature in support of debuginfo and reproducible builds
This PR adds the `-Zremap-path-prefix-from`/`-Zremap-path-prefix-to` commandline option pair and is a more general implementation of #41419. As opposed to the previous attempt, this implementation should enable reproducible builds regardless of the working directory of the compiler.
This implementation of the feature is more general in the sense that the re-mapping will affect *all* paths the compiler emits, including the ones in error messages.
r? @alexcrichton
Currently our slowest test suite on android, run-pass, takes over 5 times longer
than the x86_64 component (~400 -> ~2200s). Typically QEMU emulation does indeed
add overhead, but not 5x for this kind of workload. One of the slowest parts of
the Android process is that *compilation* happens serially. Tests themselves
need to run single-threaded on the emulator (due to how the test harness works)
and this forces the compiles themselves to be single threaded.
Now Travis gives us more than one core per machine, so it'd be much better if we
could take advantage of them! The emulator itself is still fundamentally
single-threaded, but we should see a nice speedup by sending binaries for it to
run much more quickly.
It turns out that we've already got all the tools to do this in-tree. The
qemu-test-{server,client} that are in use for the ARM Linux testing are a
perfect match for the Android emulator. This commit migrates the custom adb
management code in compiletest/rustbuild to the same qemu-test-{server,client}
implementation that ARM Linux uses.
This allows us to lift the parallelism restriction on the compiletest test
suites, namely run-pass. Consequently although we'll still basically run the
tests themselves in single threaded mode we'll be able to compile all of them in
parallel, keeping the pipeline much more full and using more cores for the work
at hand. Additionally the architecture here should be a bit speedier as it
should have less overhead than adb which is a whole new process on both the host
and the emulator!
Locally on an 8 core machine I've seen the run-pass test suite speed up from
taking nearly an hour to only taking 6 minutes. I don't think we'll see quite a
drastic speedup on Travis but I'm hoping this change can place the Android tests
well below 2 hours instead of just above 2 hours.
Because the client/server here are now repurposed for more than just QEMU,
they've been renamed to `remote-test-{server,client}`.
Note that this PR does not currently modify how debuginfo tests are executed on
Android. While parallelizable it wouldn't be quite as easy, so that's left to
another day. Thankfully that test suite is much smaller than the run-pass test
suite.
As a final fix I discovered that the ARM and Android test suites were actually
running all library unit tests (e.g. stdtest, coretest, etc) twice. I've
corrected that to only run tests once which should also give a nice boost in
overall cycle time here.
Previously the `cargotest` suite would run some arbitrary revision of Cargo's
test suite, but now that we're bundling it in tree we should be running the
Cargo submodule's test suite instead.
