ship LLVM tools with the toolchain
this PR adds llvm-{nm,objcopy,objdump,size} to the rustc sysroot (right next to LLD)
this slightly increases the size of the rustc component. I measured these numbers on x86_64 Linux:
- rustc-1.27.0-dev-x86_64-unknown-linux-gnu.tar.gz 180M -> 193M (+7%)
- rustc-1.27.0-dev-x86_64-unknown-linux-gnu.tar.xz 129M -> 137M (+6%)
r? @alexcrichton
cc #49584
add a dist builder to build rust-std components for the THUMB targets
the rust-std component only contains the core and compiler-builtins (+c +mem) crates
cc #49382
- I'm not entirely sure if this PR alone will produce rust-std components installable by rustup or if something else needs to be changed
- I could have done the THUMB builds in an existing builder / image; I wasn't sure if that was a good idea so I added a new image
- I could build other crates like alloc into the rust-std component but, AFAICT, that would require calling Cargo a second time (one for alloc and one for compiler-builtins), or have alloc depend on compiler-builtins (#49503 will perform that change) *and* have alloc resurface the "c" and "mem" Cargo features.
r? @alexcrichton
This ensures that each build will support the testing design of "dry
running" builds. It's also checked that a dry run build is equivalent
step-wise to a "wet" run build; the graphs we generate when running are
directly compared node/node and edge/edge, both for order and contents.
Allow installing rustfmt without config.extended
This assertion was preventing `./x.py install rustfmt` if attempted
without an "extended" build configuration, but it actually builds and
installs just fine.
This assertion was preventing `./x.py install rustfmt` if attempted
without an "extended" build configuration, but it actually builds and
installs just fine.
This commit refactors how the path to the linker that we're going to invoke is
selected. Previously all targets listed *both* a `LinkerFlavor` and a `linker`
(path) option, but this meant that whenever you changed one you had to change
the other. The purpose of this commit is to avoid coupling these where possible.
Target specifications now only unconditionally define the *flavor* of the linker
that they're using by default. If not otherwise specified each flavor now
implies a particular default linker to run. As a result, this means that if
you'd like to test out `ld` for example you should be able to do:
rustc -Z linker-flavor=ld foo.rs
whereas previously you had to do
rustc -Z linker-flavor=ld -C linker=ld foo.rs
This will hopefully make it a bit easier to tinker around with variants that
should otherwise be well known to work, for example with LLD, `ld` on OSX, etc.
This commit imports the LLD project from LLVM to serve as the default linker for
the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently
removed along with "binaryen linker" support in rustc.
Moving to LLD brings with it a number of benefits for wasm code:
* LLD is itself an actual linker, so there's no need to compile all wasm code
with LTO any more. As a result builds should be *much* speedier as LTO is no
longer forcibly enabled for all builds of the wasm target.
* LLD is quickly becoming an "official solution" for linking wasm code together.
This, I believe at least, is intended to be the main supported linker for
native code and wasm moving forward. Picking up support early on should help
ensure that we can help LLD identify bugs and otherwise prove that it works
great for all our use cases!
* Improvements to the wasm toolchain are currently primarily focused around LLVM
and LLD (from what I can tell at least), so it's in general much better to be
on this bandwagon for bugfixes and new features.
* Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD
will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which
means a postprocessor is no longer needed to show off Rust's "small wasm
binary size".
LLD is added in a pretty standard way to rustc right now. A new rustbuild target
was defined for building LLD, and this is executed when a compiler's sysroot is
being assembled. LLD is compiled against the LLVM that we've got in tree, which
means we're currently on the `release_60` branch, but this may get upgraded in
the near future!
LLD is placed into rustc's sysroot in a `bin` directory. This is similar to
where `gcc.exe` can be found on Windows. This directory is automatically added
to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd`
linker which implements the interface that `wasm-ld`, LLD's frontend, expects.
Like Emscripten the LLD target is currently only enabled for Tier 1 platforms,
notably OSX/Windows/Linux, and will need to be installed manually for compiling
to wasm on other platforms. LLD is by default turned off in rustbuild, and
requires a `config.toml` option to be enabled to turn it on.
Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD
has a native option for controlling this.
[gc]: https://reviews.llvm.org/D42511
Right now the `--build` option was accidentally omitted, so we're bootstraping
from `x86_64` to `i686`. In addition to being slower (more compiles) that's not
actually bootstrapping!
Right now this directory is located under:
$sysroot/lib/rustlib/$target/lib/codegen-backends
but after seeing what we do in a few other places it seems that a more
appropriate location would be:
$sysroot/lib/rustlib/$target/codegen-backends
so this commit moves it!
Building on the work of # 45684 this commit updates the compiler to
unconditionally load the `rustc_trans` crate at runtime instead of linking to it
at compile time. The end goal of this work is to implement # 46819 where rustc
will have multiple backends available to it to load.
This commit starts off by removing the `extern crate rustc_trans` from the
driver. This involved moving some miscellaneous functionality into the
`TransCrate` trait and also required an implementation of how to locate and load
the trans backend. This ended up being a little tricky because the sysroot isn't
always the right location (for example `--sysroot` arguments) so some extra code
was added as well to probe a directory relative to the current dll (the
rustc_driver dll).
Rustbuild has been updated accordingly as well to have a separate compilation
invocation for the `rustc_trans` crate and assembly it accordingly into the
sysroot. Finally, the distribution logic for the `rustc` package was also
updated to slurp up the trans backends folder.
A number of assorted fallout changes were included here as well to ensure tests
pass and such, and they should all be commented inline.
This is a forward-port of:
* 9426dda83d
* cbfb985895
from the beta branch which is used to automatically calculate the beta number
based on the number of merges to the beta branch so far.
This reverts commit ab018c76e1.
This also adds the `ToolBuild::is_ext_tool` field to replace the previous
`ToolBuild::expectation` field, to indicate whether a build-failure of
certain tool is essential.
If build failed for these tools, they will be automatically skipped from
distribution, and will not fail the whole build.
Test failures are *not* ignored, nor build failure of other tools (e.g.
cargo). Therefore it should have no observable effect to the current CI
system.
This is step 1/8 of automatic management of broken tools #45861.
rustbuild: distribute cargo-fmt alongside rustfmt
Not sure whether we want that nor if it's the right way to do so, but it feels quite weird to have rustfmt without cargo-fmt. Or are there other plans wrt that?
What do you think @nrc ?
This commit adds a new target to the compiler: wasm32-unknown-unknown. This
target is a reimagining of what it looks like to generate WebAssembly code from
Rust. Instead of using Emscripten which can bring with it a weighty runtime this
instead is a target which uses only the LLVM backend for WebAssembly and a
"custom linker" for now which will hopefully one day be direct calls to lld.
Notable features of this target include:
* There is zero runtime footprint. The target assumes nothing exists other than
the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker
is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this
target.
* Most of the standard library is stubbed out to return an error, but anything
related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new
target.
This target is currently somewhat janky due to how linking works. The "linking"
is currently unconditional whole program LTO (aka LLVM is being used as a
linker). Naturally that means compiling programs is pretty slow! Eventually
though this target should have a linker.
This target is also intended to be quite experimental. I'm hoping that this can
act as a catalyst for further experimentation in Rust with WebAssembly. Breaking
changes are very likely to land to this target, so it's not recommended to rely
on it in any critical capacity yet. We'll let you know when it's "production
ready".
---
Currently testing-wise this target is looking pretty good but isn't complete.
I've got almost the entire `run-pass` test suite working with this target (lots
of tests ignored, but many passing as well). The `core` test suite is still
getting LLVM bugs fixed to get that working and will take some time. Relatively
simple programs all seem to work though!
---
It's worth nothing that you may not immediately see the "smallest possible wasm
module" for the input you feed to rustc. For various reasons it's very difficult
to get rid of the final "bloat" in vanilla rustc (again, a real linker should
fix all this). For now what you'll have to do is:
cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm
And then `bar.wasm` should be the smallest we can get it!
---
In any case for now I'd love feedback on this, particularly on the various
integration points if you've got better ideas of how to approach them!
This commit removes the `rand` crate from the standard library facade as
well as the `__rand` module in the standard library. Neither of these
were used in any meaningful way in the standard library itself. The only
need for randomness in libstd is to initialize the thread-local keys of
a `HashMap`, and that unconditionally used `OsRng` defined in the
standard library anyway.
The cruft of the `rand` crate and the extra `rand` support in the
standard library makes libstd slightly more difficult to port to new
platforms, namely WebAssembly which doesn't have any randomness at all
(without interfacing with JS). The purpose of this commit is to clarify
and streamline randomness in libstd, focusing on how it's only required
in one location, hashmap seeds.
Note that the `rand` crate out of tree has almost always been a drop-in
replacement for the `rand` crate in-tree, so any usage (accidental or
purposeful) of the crate in-tree should switch to the `rand` crate on
crates.io. This then also has the further benefit of avoiding
duplication (mostly) between the two crates!
Make sure to clear out the stageN-{rustc,std,tools} directories.
We copy built tool binaries into a dedicated directory to avoid deleting them,
stageN-tools-bin. These aren't ever cleared out by code, since there should be
no reason to do so, and we'll simply overwrite them as necessary.
When clearing out the stageN-{std,rustc,tools} directories, make sure to delete
both Cargo directories -- per-target and build scripts. This ensures that
changing libstd doesn't cause problems due to build scripts not being rebuilt,
even though they should be.
Fixes https://github.com/rust-lang/rust/issues/44739.
