build: dist: defer PlainSourceTarball
Apparently it changes some tool sources and invalidates their fingerprints, forcing us to build them several times (before and after vendoring sources).
I have not dug into why vendoring actually invalidates the figreprints, but moving the vendoring lower in the pipeline seems to avoid the issue.
I could imagine that we somehow write a .cargo/config somewhere which somehow makes subsequent builds use the vendored deps but I was not able to find anything.
I checked the sizes of generated archives pre and post patch and their are the same, so I hope there is no functional change.
Fixes#93033
x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.
As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:
./x.py test --exclude library/std
...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).
This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:
./x.py test --exclude test::library/std
This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:
| `--exclude` | Docs | Tests |
| ------------------- | ------- | ------- |
| `library/std` | Skipped | Skipped |
| `doc::library/std` | Skipped | Run |
| `test::library/std` | Run | Skipped |
Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.
Building a dozen separate regexps for each test in compiletest consumes significant amounts of CPU cycles.
Using `RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER` stabilizes hashes calcuated for the individual tests so
no test-dependent normalization is needed. Hashes for the standard library still change so some
normalizations are still needed.
Add manifest docs fallback.
This adds a fallback so that the rustup manifest will contain the rust-docs component for all hosts. There is a mapping so that the docs that get downloaded are roughly close to the actual host. There inevitably will be things that don't match. Ideally the standard library docs would be the same for every platform (`cfg(doc)` goes a long way towards this), but there are still lots of minor differences.
Closes#69525
Apparently it changes some tool sources and invalidates their fingerprints, forcing us to build them several times (before and after vendoring sources).
I have not dug into why vendoring actually invalidates the figreprints, but the moving the vendoring lower in the pipeline seems to avoid the issue.
I could imagine that we somehow write a .cargo/config somewhere which somehow makes subsequent builds use the vendored deps but I was not able to find anything.
I checked the sizes of generated archives pre and post patch and their are the same, so I hope there is not functional change.
Fixes#93033
This builder is the slowest in the fleet. This should cut a considerable
amount of time. The manifest should now include the docs from
x86_64-apple-darwin. Although those docs are slightly different, it
should be close enough. When aarch64-apple-darwin heads towards tier 1,
we can revisit whether or not to re-enable the docs.
Exclude llvm-libunwind from the self-contained set on s390x-musl targets
llvm-libunwind does not support s390x targets at present, so we cannot build it
for s390x targets. Accordingly, remove it from the self-contained set.
Break the loop
A missing break statement lead to an infinite loop in bootstrap.py.
I also added a short sleep so it's not constantly running at 100%. But I can remove that if it's not wanted.
Fixes#76661
Currently the output of a command like `./x.py build --stage 0 library/std` is this:
```
Updating only changed submodules
Submodules updated in 0.02 seconds
extracting [...]
Compiling [...]
Finished dev [unoptimized] target(s) in 17.53s
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Compling [...]
Finished release [optimized + debuginfo] target(s) in 21.99s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Build completed successfully in 0:00:51
```
I find the part before the "Building stage0 std artifacts" a bit confusing.
After this commit, it looks like this:
```
Updating only changed submodules
Submodules updated in 0.02 seconds
extracting [...]
Building rustbuild
Compiling [...]
Finished dev [unoptimized] target(s) in 17.53s
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Compling [...]
Finished release [optimized + debuginfo] target(s) in 21.99s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Build completed successfully in 0:00:51
```
The "Building rustbuild" label makes it clear what the first cargo build
invocation is for. The indentation of the "Submodules updated" line
indicates it is a sub-step of a parent task.
This slightly improves compilation time by reducing linking time
(saving about a 1/10 of the the total compilation time after
changing rustbuild) and slightly reduces disk usage (from 16MB for
the rustc wrapper to 4MB).
The task of the macro is simple enough that a decl macro is almost ten
times shorter than the original proc macro. The proc macro is 159 lines
while the decl macro is just 18 lines.
This reduces the amount of dependencies of rustbuild from 45 to 37. It
also slight reduces compilation time from 47s to 44s for debug builds.
Lock bootstrap (x.py) build directory
Closes#76661, closes#80849,
`x.py` creates a lock file at `project_root/lock.db`
r? `@jyn514` , because he was one that told me about this~
Prevent spurious build failures and other bugs caused by parallel runs of
x.py. We back the lock with sqlite, so that we have a cross-platform locking
strategy, and which can be done nearly first in the build process (from Python),
which helps move the lock as early as possible.
DF_ORIGIN flag signifies that the object being loaded may make reference to the $ORIGIN substitution string.
Some implementations are just ignoring DF_ORIGIN and do substitution for $ORIGIN if present (whatever DF_ORIGIN pr
Set the flag inconditionally if rpath is wanted.
On Windows, `x.py dist` command panics without proper error message if 'WIX' environment variable is not set. This patch changes `Option::unwrap()` to `Option::expect()`.
This already enables `download-rustc`, so it's quick to build rustdoc,
and this makes it less confusing when changes to rustdoc aren't reflected in the docs.
Note that this uses 2 and not 1 because `download-rustc` only affects stage 2 runs.
Some projects (e.g. the `bootimage` crate) may require the
user to install the `llvm-tools-preview` rustup component.
However, this cannot be easily done with a locally built toolchain.
To allow a local toolchain to be used a drop-in replacement for
a normal rustup toolchain in more cases, this PR copies the built
LLVM tools to the sysoot. From the perspective a tool looking
at the sysroot, this is equivalent to installing `llvm-tools-preview`.
Add support for riscv64gc-unknown-freebsd
For https://doc.rust-lang.org/nightly/rustc/target-tier-policy.html#tier-3-target-policy:
* A tier 3 target must have a designated developer or developers (the "target maintainers") on record to be CCed when issues arise regarding the target. (The mechanism to track and CC such developers may evolve over time.)
For all Rust targets on FreeBSD, it's [rust@FreeBSD.org](mailto:rust@FreeBSD.org).
* Targets must use naming consistent with any existing targets; for instance, a target for the same CPU or OS as an existing Rust target should use the same name for that CPU or OS. Targets should normally use the same names and naming conventions as used elsewhere in the broader ecosystem beyond Rust (such as in other toolchains), unless they have a very good reason to diverge. Changing the name of a target can be highly disruptive, especially once the target reaches a higher tier, so getting the name right is important even for a tier 3 target.
Done.
* Target names should not introduce undue confusion or ambiguity unless absolutely necessary to maintain ecosystem compatibility. For example, if the name of the target makes people extremely likely to form incorrect beliefs about what it targets, the name should be changed or augmented to disambiguate it.
Done
* Tier 3 targets may have unusual requirements to build or use, but must not create legal issues or impose onerous legal terms for the Rust project or for Rust developers or users.
Done.
* The target must not introduce license incompatibilities.
Done.
* Anything added to the Rust repository must be under the standard Rust license (MIT OR Apache-2.0).
Fine with me.
* The target must not cause the Rust tools or libraries built for any other host (even when supporting cross-compilation to the target) to depend on any new dependency less permissive than the Rust licensing policy. This applies whether the dependency is a Rust crate that would require adding new license exceptions (as specified by the tidy tool in the rust-lang/rust repository), or whether the dependency is a native library or binary. In other words, the introduction of the target must not cause a user installing or running a version of Rust or the Rust tools to be subject to any new license requirements.
Done.
* If the target supports building host tools (such as rustc or cargo), those host tools must not depend on proprietary (non-FOSS) libraries, other than ordinary runtime libraries supplied by the platform and commonly used by other binaries built for the target. For instance, rustc built for the target may depend on a common proprietary C runtime library or console output library, but must not depend on a proprietary code generation library or code optimization library. Rust's license permits such combinations, but the Rust project has no interest in maintaining such combinations within the scope of Rust itself, even at tier 3.
Done.
* Targets should not require proprietary (non-FOSS) components to link a functional binary or library.
Done.
* "onerous" here is an intentionally subjective term. At a minimum, "onerous" legal/licensing terms include but are not limited to: non-disclosure requirements, non-compete requirements, contributor license agreements (CLAs) or equivalent, "non-commercial"/"research-only"/etc terms, requirements conditional on the employer or employment of any particular Rust developers, revocable terms, any requirements that create liability for the Rust project or its developers or users, or any requirements that adversely affect the livelihood or prospects of the Rust project or its developers or users.
Fine with me.
* Neither this policy nor any decisions made regarding targets shall create any binding agreement or estoppel by any party. If any member of an approving Rust team serves as one of the maintainers of a target, or has any legal or employment requirement (explicit or implicit) that might affect their decisions regarding a target, they must recuse themselves from any approval decisions regarding the target's tier status, though they may otherwise participate in discussions.
Ok.
* This requirement does not prevent part or all of this policy from being cited in an explicit contract or work agreement (e.g. to implement or maintain support for a target). This requirement exists to ensure that a developer or team responsible for reviewing and approving a target does not face any legal threats or obligations that would prevent them from freely exercising their judgment in such approval, even if such judgment involves subjective matters or goes beyond the letter of these requirements.
Ok.
* Tier 3 targets should attempt to implement as much of the standard libraries as possible and appropriate (core for most targets, alloc for targets that can support dynamic memory allocation, std for targets with an operating system or equivalent layer of system-provided functionality), but may leave some code unimplemented (either unavailable or stubbed out as appropriate), whether because the target makes it impossible to implement or challenging to implement. The authors of pull requests are not obligated to avoid calling any portions of the standard library on the basis of a tier 3 target not implementing those portions.
std is implemented.
* The target must provide documentation for the Rust community explaining how to build for the target, using cross-compilation if possible. If the target supports running tests (even if they do not pass), the documentation must explain how to run tests for the target, using emulation if possible or dedicated hardware if necessary.
Building is possible the same way as other Rust on FreeBSD targets.
* Tier 3 targets must not impose burden on the authors of pull requests, or other developers in the community, to maintain the target. In particular, do not post comments (automated or manual) on a PR that derail or suggest a block on the PR based on a tier 3 target. Do not send automated messages or notifications (via any medium, including via `@)` to a PR author or others involved with a PR regarding a tier 3 target, unless they have opted into such messages.
Ok.
* Backlinks such as those generated by the issue/PR tracker when linking to an issue or PR are not considered a violation of this policy, within reason. However, such messages (even on a separate repository) must not generate notifications to anyone involved with a PR who has not requested such notifications.
Ok.
* Patches adding or updating tier 3 targets must not break any existing tier 2 or tier 1 target, and must not knowingly break another tier 3 target without approval of either the compiler team or the maintainers of the other tier 3 target.
Ok.
* In particular, this may come up when working on closely related targets, such as variations of the same architecture with different features. Avoid introducing unconditional uses of features that another variation of the target may not have; use conditional compilation or runtime detection, as appropriate, to let each target run code supported by that target.
Ok.
Previously, it would concatenate the relative path to the current
subdirectory, which looked at the wrong folder.
I tested this by checking out `1.56.1`, changing the current directory
to `src/`, and running `../x.py build`.
bootstap: create .cargo/config only if not present
In some situations we should want on influence into the .cargo/config
when we use vendored source. One example is #90764, when we want to
workaround some references to crates forked and living in git, that are
missing in the vendor/ directory.
This commit will create the .cargo/config file only when the .cargo/
directory needs to be created.
std: Get the standard library compiling for wasm64
This commit goes through and updates various `#[cfg]` as appropriate to
get the wasm64-unknown-unknown target behaving similarly to the
wasm32-unknown-unknown target. Most of this is just updating various
conditions for `target_arch = "wasm32"` to also account for `target_arch
= "wasm64"` where appropriate. This commit also lists `wasm64` as an
allow-listed architecture to not have the `restricted_std` feature
enabled, enabling experimentation with `-Z build-std` externally.
The main goal of this commit is to enable playing around with
`wasm64-unknown-unknown` externally via `-Z build-std` in a way that's
similar to the `wasm32-unknown-unknown` target. These targets are
effectively the same and only differ in their pointer size, but wasm64
is much newer and has much less ecosystem/library support so it'll still
take time to get wasm64 fully-fledged.
