When cross compiling on macOS with `llvm.link-shared` enabled,
the symlink creation will fail after compiling LLVM for the target
architecture, because it will attempt to create the symlink in the
host LLVM directory, which was already created when being built.
This commit changes the symlink path to the actual LLVM output.
Fix `x dist rust-dev` on a fresh checkout
Previously, it required you to manually run `x build` first, because it
assumed the LLVM binaries were already present.
bootstrap: illumos platform flags for split-debuginfo
Bootstrap currently provides `-Zunstable-options` for platforms
when using split debuginfo - this commit adds it for the illumos
target too.
Allow macOS to build LLVM as shared library
Inspired by how [homebrew](https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/llvm.rb) builds and distributes llvm, here we manually create a symlink with a versioned dylib path to make `llvm-config` work properly. Note, the resulting `rustc` executable and `librustc_driver-<hash>.dylib` still links to the un-versioned `libLLVM.dylib` as expected when distributed in the final output. I have confirmed this by checking `otool -L` on both binaries.
After the change, enabling `llvm.link-shared` and `llvm.thin-lto` will be possible on macOS.
Add a `--build-dir` flag to rustbuild
This adds an optional `--build-dir <path>` flag to rustbuild (to both the python and rust code in src/bootstrap). If provided, it overrides build directory from the config file (if any was provided).
My reason for wanting this is that I often will make a change, save, and then go run `x.py check` or `x.py test` (or something). Because I've saved, vscode will start doing its thing in the background, but this will take the file lock, preventing `x.py` from running until vscode finishes whatever it's doing (since the manually invoked x.py won't be able to acquire said file lock). This is annoying, because I'd rather the command I explicitly invoke *not* wait for r-a to complete, as r-a's check is conceptually a background task (and one which can take quite some time to complete).
Anyway, while there are likely other ways this could be handled, if you have the disk space an easy way is to just have vscode be configured to use a different build directory, and then they never have to block each-other.
This can currently be arranged without this patch, by maintaining two `config.toml`s, one of which has a different build dir, and just exists to be passed into the overridden check command in vscode.
Unfortunately, this has the downside of requiring I maintain two `config.toml`s and keep them (at least somewhat) in sync, aside from the build dir. I dislike for several reasons, not the least of which because I know myself well enough to know that these will inevitably get out of sync and confuse me in the future (perhaps this case would be different since I've thought about it enough to write this patch? Who knows, I'd rather not find out).
Either way, it would be much easier for me to have a way for *only* the build directory to differ, which this patch provides by way of a new flag. I suggested this to `@jyn514` who indicated it sounded reasonable so long as it didn't add too much complexity, which I think I've achieved, but he can be the judge.
Anyway, with this patch I can just use something like `["python3", "x.py", "check", "--build-dir", "build-vscode", "--json-output"]` as the overridden check command to rust-analyzer, and do not need to futz with any additional `config.toml`s. Which is very nice!
I've tested this manually, and can confirm that it works. I'm not sure if it needs automated tests, or where I should add them if so.
r? `@jyn514` (who has had to put up with my complaints about this... many times. <3)
Let rust-analyzer ship on stable, non-preview
The consensus on rust-lang/rust-analyzer#12432 seems to be that we are ready for `rust-analyzer` to ship as a rustup component on the beta and stable channels. This won't always be the preferred distribution method, e.g. the VS Code extension will probably still independently update to its weekly releases, but it's still useful to have a component that follows the release train with the rest of the Rust toolchain. So this removes the nightly-only gating on the bundled component, and removes the "-preview" suffix as well by the usual renaming mechanism.
cc ``@rust-lang/wg-rls-2`` ``@rust-lang/release``
Ensure that `static_crt` is set in the bootstrapper whenever using `cc-rs` to get a compiler command line.
When attempting to build rustc with LLVM on Windows, I noticed that the CRT flag provided to the C and C++ Compilers was inconsistent:
```
"-DCMAKE_C_FLAGS=-nologo -MT -Brepro" "-DCMAKE_CXX_FLAGS=-nologo -MD -Brepro"
```
Since the bootstrapper also sets the various `LLVM_USE_CRT` variables, this resulted in cl.exe reporting a bunch of warnings:
```
cl : Command line warning D9025 : overriding '/MD' with '/MT'
```
The root cause for this is that `cc_detect::find` was creating a `cc::Build` twice, but didn't set `static_crt` the second time.
It's possible that this what is also causing #81381
When attempting to build rustc with LLVM on Windows, I noticed that the CRT flag provided to the C and C++ Compilers was inconsistent:
```
"-DCMAKE_C_FLAGS=-nologo -MT -Brepro" "-DCMAKE_CXX_FLAGS=-nologo -MD -Brepro"
```
Since the bootstrapper also sets the various `LLVM_USE_CRT` variables, this resulted in cl.exe reporting a bunch of warnings:
```
cl : Command line warning D9025 : overriding '/MD' with '/MT'
```
The root cause for this is that `cc_detect::find` was creating a `cc::Build` twice, but didn't set `static_crt` the second time.
It's possible that this what is also causing #81381
Fix LLVM rebuild with download-ci-llvm.
This fixes an issue where updating a local checkout that includes a change in `src/version` causes a linking failure.
The cause is that the `rustc_llvm` build script uses `rerun-if-changed` of `llvm-config` to know if it needs to rerun. Cargo only compares the timestamp of the last time the build script to the file. However, extracting the tar files retains the timestamps in the tarball which may be some time in the past. Since `src/version` is included in the LLVM `.so` filename, `rustc` attempts to load the wrong shared library since the `rustc_llvm` build script doesn't rerun.
https://github.com/rust-lang/cargo/issues/10791 contains a more detailed explanation.
The solution here is a hack which updates the timestamp of `llvm-config` to the current time when it is extracted.
This is a bit of a hack, but seems to be the best solution I can think of until https://github.com/rust-lang/cargo/issues/10791 is fixed. There are likely several other situations where this is a problem (such as using system LLVM), and this isn't really a complete fix.
Note that apple platforms are not directly affected by this problem because they don't have a version in the dylib filename.
How to test this:
1. On a linux host, enable download-ci-llvm
2. Check out 7036449c77 (the commit just before the last version bump)
3. `./x.py build library/std`
4. Check out 5f015a24f9 (the commit that bumped the version)
5. `./x.py build library/std`
Fixes#98495
This is difficult to support without submodule handling in bootstrap.py, because cargo will refuse
to vendor sources unless it knows the Cargo.toml files of all tools in tree. Moving vendor support
to rustbuild means that rustbuild will be built without vendoring.
Rather than trying to solve this, just remove support altogether and require
people to use `rustc-src` if they want vendoring (or run `cargo vendor` manually).
These submodules were previously updated in python because Cargo gives a hard error if toml files
are missing from the workspace:
```
error: failed to load manifest for workspace member `/home/jnelson/rust-lang/rust/src/tools/rls`
Caused by:
failed to read `/home/jnelson/rust-lang/rust/src/tools/rls/Cargo.toml`
Caused by:
No such file or directory (os error 2)
failed to run: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/jnelson/rust-lang/rust/src/bootstrap/Cargo.toml
```
However, bootstrap doesn't actually need to be part of the workspace.
Remove it so we can move submodule handling fully to Rust, avoiding duplicate code between Rust and Python.
Note that this does break `cargo run`; it has to be `cd src/bootstrap && cargo run` now.
Given that we're planning to make the main entrypoint a shell script (or rust binary),
I think this is a good tradeoff for reduced complexity in bootstrap.py.
Make "Assemble stage1 compiler" orders of magnitude faster (take 2)
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
- Panic if we generate a symbolic link in a tarball
- Change install to use copy internally, like in my previous PR
- Change copy to dereference symbolic links, which avoids the previous regression in #96803.
I also took the liberty of fixing `x dist llvm-tools` to work even if you don't call `x build` previously.
This used to take upwards of 5 seconds for me locally. I found that the
culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change
it to use `copy()` internally, which uses hard links instead when
available.
Note that this has a change in behavior: Installing a file will also
change permissions on the source, not just the destination, if hard
links are used.
To avoid changing the behavior on symlinks for existing code, I
introduce a new function `copy_internal` which only dereferences
symlinks when told to do so.
Don't build the compiler before building rust-demangler
This saves a lot of time compiling, since rust-demangler doesn't actually use any unstable features.
This is not quite ideal because it uses ToolStd, not ToolBootstrap, so rust-demangler would be able to add unstable library features in the future. But it's a lot better than before, and `builder.cargo` doesn't currently know how to handle stages other than 0.
Pass all paths to `Step::run` at once when using `ShouldRun::krate`
Helps with https://github.com/rust-lang/rust/pull/95503. The goal is to run `cargo test -p rustc_data_structures -p rustc_lint_defs` instead of `cargo test -p rustc_data_structures; cargo test -p rustc_lint_defs`, which should both recompile less and avoid replaying cached warnings.
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
`@rustbot` label +A-rustbuild
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
Allow configuring where artifacts are downloaded from
Bootstrap has support for downloading prebuilt LLVM and rustc artifacts to speed up local builds, but that currently works only for users working on `rust-lang/rust`. Forks of the repository (for example Ferrocene) might have different URLs to download artifacts from, or might use a different email address on merge commits, breaking both LLVM and rustc artifact downloads.
This PR refactors bootstrap to load the download URLs and other constants from `src/stage0.json`, allowing downstream forks to tweak those values. It also future-proofs the download code to easily allow forks to add their own custom protocols (like `s3://`).
This PR is best reviewed commit-by-commit.
As was discovered in https://github.com/rust-lang/rust/pull/93628#issuecomment-1154697627 ,
adding #[cfg(bootstrap)] to a rust-internal proc macro crate
would yield an unexpected cfg name error, at least on later
stages wher the bootstrap cfg arg wasn't set.
rustc already passes arguments to mark bootstrap as expected,
however the means of delivery through the RUSTFLAGS env var
is unable to reach proc macro crates, as described
in the issue linked in the code this commit touches.
This wouldn't be an issue for cfg args that get passed through
RUSTFLAGS, as they would never become *active* either, so
any usage of one of these flags in a proc macro's code would
legitimately yield a lint warning. But since dc302587e2,
rust takes extra measures to pass --cfg=bootstrap even in
proc macros, by passing it via the wrapper. Thus, we need
to send the flags to mark bootstrap as expected also from the
wrapper, so that #[cfg(bootstrap)] also works from proc macros.
I want to thank Urgau and jplatte for helping me find the cause of this. ❤️
Add Apple WatchOS compile targets
Hello,
I would like to add the following target triples for Apple WatchOS as Tier 3 platforms:
armv7k-apple-watchos
arm64_32-apple-watchos
x86_64-apple-watchos-sim
There are some pre-requisites Pull Requests:
https://github.com/rust-lang/compiler-builtins/pull/456 (merged)
https://github.com/alexcrichton/cc-rs/pull/662 (pending)
https://github.com/rust-lang/libc/pull/2717 (merged)
There will be a subsequent PR with standard library changes for WatchOS. Previous compiler and library changes were in a single PR (https://github.com/rust-lang/rust/pull/94736) which is now closed in favour of separate PRs.
Many thanks!
Vlad.
### Tier 3 Target Requirements
Adds support for Apple WatchOS compile targets.
Below are details on how this target meets the requirements for tier 3:
> tier 3 target must have a designated developer or developers (the "target maintainers") on record to be CCed when issues arise regarding the target. (The mechanism to track and CC such developers may evolve over time.)
`@deg4uss3r` has volunteered to be the target maintainer. I am also happy to help if a second maintainer is required.
> Targets must use naming consistent with any existing targets; for instance, a target for the same CPU or OS as an existing Rust target should use the same name for that CPU or OS. Targets should normally use the same names and naming conventions as used elsewhere in the broader ecosystem beyond Rust (such as in other toolchains), unless they have a very good reason to diverge. Changing the name of a target can be highly disruptive, especially once the target reaches a higher tier, so getting the name right is important even for a tier 3 target.
Uses the same naming as the LLVM target, and the same convention as other Apple targets.
> Target names should not introduce undue confusion or ambiguity unless absolutely necessary to maintain ecosystem compatibility. For example, if the name of the target makes people extremely likely to form incorrect beliefs about what it targets, the name should be changed or augmented to disambiguate it.
I don't believe there is any ambiguity here.
> Tier 3 targets may have unusual requirements to build or use, but must not create legal issues or impose onerous legal terms for the Rust project or for Rust developers or users.
I don't see any legal issues here.
> The target must not introduce license incompatibilities.
> Anything added to the Rust repository must be under the standard Rust license (MIT OR Apache-2.0).
> The target must not cause the Rust tools or libraries built for any other host (even when supporting cross-compilation to the target) to depend on any new dependency less permissive than the Rust licensing policy. This applies whether the dependency is a Rust crate that would require adding new license exceptions (as specified by the tidy tool in the rust-lang/rust repository), or whether the dependency is a native library or binary. In other words, the introduction of the target must not cause a user installing or running a version of Rust or the Rust tools to be subject to any new license requirements.
> If the target supports building host tools (such as rustc or cargo), those host tools must not depend on proprietary (non-FOSS) libraries, other than ordinary runtime libraries supplied by the platform and commonly used by other binaries built for the target. For instance, rustc built for the target may depend on a common proprietary C runtime library or console output library, but must not depend on a proprietary code generation library or code optimization library. Rust's license permits such combinations, but the Rust project has no interest in maintaining such combinations within the scope of Rust itself, even at tier 3.
> Targets should not require proprietary (non-FOSS) components to link a functional binary or library.
> "onerous" here is an intentionally subjective term. At a minimum, "onerous" legal/licensing terms include but are not limited to: non-disclosure requirements, non-compete requirements, contributor license agreements (CLAs) or equivalent, "non-commercial"/"research-only"/etc terms, requirements conditional on the employer or employment of any particular Rust developers, revocable terms, any requirements that create liability for the Rust project or its developers or users, or any requirements that adversely affect the livelihood or prospects of the Rust project or its developers or users.
I see no issues with any of the above.
> Neither this policy nor any decisions made regarding targets shall create any binding agreement or estoppel by any party. If any member of an approving Rust team serves as one of the maintainers of a target, or has any legal or employment requirement (explicit or implicit) that might affect their decisions regarding a target, they must recuse themselves from any approval decisions regarding the target's tier status, though they may otherwise participate in discussions.
> This requirement does not prevent part or all of this policy from being cited in an explicit contract or work agreement (e.g. to implement or maintain support for a target). This requirement exists to ensure that a developer or team responsible for reviewing and approving a target does not face any legal threats or obligations that would prevent them from freely exercising their judgment in such approval, even if such judgment involves subjective matters or goes beyond the letter of these requirements.
Only relevant to those making approval decisions.
> Tier 3 targets should attempt to implement as much of the standard libraries as possible and appropriate (core for most targets, alloc for targets that can support dynamic memory allocation, std for targets with an operating system or equivalent layer of system-provided functionality), but may leave some code unimplemented (either unavailable or stubbed out as appropriate), whether because the target makes it impossible to implement or challenging to implement. The authors of pull requests are not obligated to avoid calling any portions of the standard library on the basis of a tier 3 target not implementing those portions.
core and alloc can be used. std support will be added in a subsequent PR.
> The target must provide documentation for the Rust community explaining how to build for the target, using cross-compilation if possible. If the target supports running tests (even if they do not pass), the documentation must explain how to run tests for the target, using emulation if possible or dedicated hardware if necessary.
Use --target=<target> option to cross compile, just like any target. Tests can be run using the WatchOS simulator (see https://developer.apple.com/documentation/xcode/running-your-app-in-the-simulator-or-on-a-device).
> Tier 3 targets must not impose burden on the authors of pull requests, or other developers in the community, to maintain the target. In particular, do not post comments (automated or manual) on a PR that derail or suggest a block on the PR based on a tier 3 target. Do not send automated messages or notifications (via any medium, including via `@)` to a PR author or others involved with a PR regarding a tier 3 target, unless they have opted into such messages.
> Backlinks such as those generated by the issue/PR tracker when linking to an issue or PR are not considered a violation of this policy, within reason. However, such messages (even on a separate repository) must not generate notifications to anyone involved with a PR who has not requested such notifications.
I don't foresee this being a problem.
> Patches adding or updating tier 3 targets must not break any existing tier 2 or tier 1 target, and must not knowingly break another tier 3 target without approval of either the compiler team or the maintainers of the other tier 3 target.
> In particular, this may come up when working on closely related targets, such as variations of the same architecture with different features. Avoid introducing unconditional uses of features that another variation of the target may not have; use conditional compilation or runtime detection, as appropriate, to let each target run code supported by that target.
No other targets should be affected by the pull request.
This commit allows users to change the contents of the "config" key in
src/stage0.json without having it overridden the next time the
bump-stage0 tool is executed.
