Autodiff Upstreaming - enzyme backend
Tracking issue: https://github.com/rust-lang/rust/issues/124509
Part of https://github.com/rust-lang/rust/pull/129175
This PR should allow building Enzyme from source on Tier 1 targets (when also building LLVM), except MSVC.
It's only a small fraction (~200 lines) of the whole upstream PR, but due to bootstrapping and the number of configurations in which rustc can be build I assume that this will be the hardest to merge, so I'm starting with it.
Happy to hear what changes are required to be able to upstream this code.
**Content:**
It contains a new configure flag `--enable-llvm-enzyme`, and will build the new Enzyme submodule when it is set.
**Discussion:**
Apparently Rust CI isn't able to clone repositories outside the rust-lang org? At least I'm seeing this error in CI:
```
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
```
Does that mean we would need to mirror github.com/EnzymeAD/Enzyme in rust-lang, until LLVM upgrades Enzyme from an Incubator project to something that ships as part of the monorepo?
Tracking:
- https://github.com/rust-lang/rust/issues/124509
Make `./x.py <cmd> compiler/<crate>` aware of the crate's features
Does not fix https://github.com/rust-lang/rust/issues/129727 on its own as the way the parallel-compiler cfg and feature flags are setup being generally incompatible with `resolver = 2` but it progresses on the issue. But this should in theory allow compiler crates to work that do not depend on the parallel compiler stuff (so some leaf crates).
forward linker option to lint-docs
This fixes an error found when building the doc for a cross-built toolchain.
```
warning: the code example in lint `unstable_syntax_pre_expansion` in /buildroots/chenx97/rustc-1.80.1-src/compiler/rustc_lint_defs/src/builtin.rs failed to generate the expected output: did not find lint `unstable_syntax_p
re_expansion` in output of example, got:
error: linking with `cc` failed: exit status: 1
...
```
Closes: #129956
copy rustc rustlib artifacts from ci-rustc
We recently (since https://github.com/rust-lang/rust/pull/129311) had an issue because some rustlib files were missing (like: "error[E0463]: can't find crate for rustc_ast") when building tools that rely on rustc. This patch fixes that by copying those files as required.
r? Kobzol
Blocker for https://github.com/rust-lang/rust/pull/122709
bootstrap: Try to track down why `initial_libdir` sometimes fails
When I try to run `x` commands from the command-line, I occasionally see a mysterious failure that looks something like this:
```text
thread 'main' panicked at src/lib.rs:341:14:
called `Result::unwrap()` on an `Err` value: StripPrefixError(())
```
It happens often enough to be annoying, but rarely enough that I can't reproduce it at will. The error message points to a particular `unwrap` call, but doesn't include enough context to determine *why* the failure occurs.
Re-running the command almost always works, so I suspect some kind of filesystem race condition (possibly involving VSCode invoking bootstrap at the same time), but there's not much I can do with the information I currently have.
So this PR includes some relevant information in the panic message when the failure occurs, in the hope that doing so will make the cause easier to track down when the failure occurs again.
Add target support for RTEMS Arm
# `armv7-rtems-eabihf`
This PR adds a new target for the RTEMS RTOS. To get things started it focuses on Xilinx/AMD Zynq-based targets, but in theory it should also support other armv7-based board support packages in the future.
Given that RTEMS has support for many POSIX functions it is mostly enabling corresponding unix features for the new target.
I also previously started a PR in libc (https://github.com/rust-lang/libc/pull/3561) to add the needed OS specific C-bindings and was told that a PR in this repo is needed first. I will update the PR to the newest version after approval here.
I will probably also need to change one line in the backtrace repo.
Current status is that I could compile rustc for the new target locally (with the updated libc and backtrace) and could compile binaries, link, and execute a simple "Hello World" RTEMS application for the target hardware.
> A proposed target or target-specific patch that substantially changes code shared with other targets (not just target-specific code) must be reviewed and approved by the appropriate team for that shared code before acceptance.
There should be no breaking changes for existing targets. Main changes are adding corresponding `cfg` switches for the RTEMS OS and adding the C binding in libc.
# Tier 3 target policy
> - A tier 3 target must have a designated developer or developers (the "target maintainers") on record to be CCed when issues arise regarding the target. (The mechanism to track and CC such developers may evolve over time.)
I will do the maintenance (for now) further members of the RTEMS community will most likely join once the first steps have been done.
> - Targets must use naming consistent with any existing targets; for instance, a target for the same CPU or OS as an existing Rust target should use the same name for that CPU or OS. Targets should normally use the same names and naming conventions as used elsewhere in the broader ecosystem beyond Rust (such as in other toolchains), unless they have a very good reason to diverge. Changing the name of a target can be highly disruptive, especially once the target reaches a higher tier, so getting the name right is important even for a tier 3 target.
> - Target names should not introduce undue confusion or ambiguity unless absolutely necessary to maintain ecosystem compatibility. For example, if the name of the target makes people extremely likely to form incorrect beliefs about what it targets, the name should be changed or augmented to disambiguate it.
> - If possible, use only letters, numbers, dashes and underscores for the name. Periods (`.`) are known to cause issues in Cargo.
The proposed triple is `armv7-rtems-eabihf`
> - Tier 3 targets may have unusual requirements to build or use, but must not create legal issues or impose onerous legal terms for the Rust project or for Rust developers or users.
> - The target must not introduce license incompatibilities.
> - Anything added to the Rust repository must be under the standard Rust license (`MIT OR Apache-2.0`).
> - The target must not cause the Rust tools or libraries built for any other host (even when supporting cross-compilation to the target) to depend on any new dependency less permissive than the Rust licensing policy. This applies whether the dependency is a Rust crate that would require adding new license exceptions (as specified by the `tidy` tool in the rust-lang/rust repository), or whether the dependency is a native library or binary. In other words, the introduction of the target must not cause a user installing or running a version of Rust or the Rust tools to be subject to any new license requirements.
> - Compiling, linking, and emitting functional binaries, libraries, or other code for the target (whether hosted on the target itself or cross-compiling from another target) must not depend on proprietary (non-FOSS) libraries. Host tools built for the target itself may depend on the ordinary runtime libraries supplied by the platform and commonly used by other applications built for the target, but those libraries must not be required for code generation for the target; cross-compilation to the target must not require such libraries at all. For instance, `rustc` built for the target may depend on a common proprietary C runtime library or console output library, but must not depend on a proprietary code generation library or code optimization library. Rust's license permits such combinations, but the Rust project has no interest in maintaining such combinations within the scope of Rust itself, even at tier 3.
> - "onerous" here is an intentionally subjective term. At a minimum, "onerous" legal/licensing terms include but are _not_ limited to: non-disclosure requirements, non-compete requirements, contributor license agreements (CLAs) or equivalent, "non-commercial"/"research-only"/etc terms, requirements conditional on the employer or employment of any particular Rust developers, revocable terms, any requirements that create liability for the Rust project or its developers or users, or any requirements that adversely affect the livelihood or prospects of the Rust project or its developers or users.
The tools consists of the cross-compiler toolchain (gcc-based). The RTEMS kernel (BSD license) and parts of the driver stack of FreeBSD (BSD license). All tools are FOSS and publicly available here: https://gitlab.rtems.org/rtems
There are also no new features or dependencies introduced to the Rust code.
> - Neither this policy nor any decisions made regarding targets shall create any binding agreement or estoppel by any party. If any member of an approving Rust team serves as one of the maintainers of a target, or has any legal or employment requirement (explicit or implicit) that might affect their decisions regarding a target, they must recuse themselves from any approval decisions regarding the target's tier status, though they may otherwise participate in discussions.
N/A to me. I am not a reviewer nor Rust team member.
> - Tier 3 targets should attempt to implement as much of the standard libraries as possible and appropriate (`core` for most targets, `alloc` for targets that can support dynamic memory allocation, `std` for targets with an operating system or equivalent layer of system-provided functionality), but may leave some code unimplemented (either unavailable or stubbed out as appropriate), whether because the target makes it impossible to implement or challenging to implement. The authors of pull requests are not obligated to avoid calling any portions of the standard library on the basis of a tier 3 target not implementing those portions.
`core` and `std` compile. Some advanced features of the `std` lib might not work yet. However, the goal of this tier 3 target it to make it easier for other people to build and run test applications to better identify the unsupported features and work towards enabling them.
> - The target must provide documentation for the Rust community explaining how to build for the target, using cross-compilation if possible. If the target supports running binaries, or running tests (even if they do not pass), the documentation must explain how to run such binaries or tests for the target, using emulation if possible or dedicated hardware if necessary.
Building is described in platform support doc. Running simple unit tests works. Running the test suite of the stdlib is currently not that easy. Trying to work towards that after the this target has been added to the nightly.
> - Tier 3 targets must not impose burden on the authors of pull requests, or other developers in the community, to maintain the target. In particular, do not post comments (automated or manual) on a PR that derail or suggest a block on the PR based on a tier 3 target. Do not send automated messages or notifications (via any medium, including via ````@`)``` to a PR author or others involved with a PR regarding a tier 3 target, unless they have opted into such messages.
Understood.
> - Backlinks such as those generated by the issue/PR tracker when linking to an issue or PR are not considered a violation of this policy, within reason. However, such messages (even on a separate repository) must not generate notifications to anyone involved with a PR who has not requested such notifications.
Ok
> - Patches adding or updating tier 3 targets must not break any existing tier 2 or tier 1 target, and must not knowingly break another tier 3 target without approval of either the compiler team or the maintainers of the other tier 3 target.
> - In particular, this may come up when working on closely related targets, such as variations of the same architecture with different features. Avoid introducing unconditional uses of features that another variation of the target may not have; use conditional compilation or runtime detection, as appropriate, to let each target run code supported by that target.
I think, I didn't add any breaking changes for any existing targets (see the comment regarding features above).
> - Tier 3 targets must be able to produce assembly using at least one of rustc's supported backends from any host target.
Can produce assembly code via the llvm backend (tested on Linux).
>
> If a tier 3 target stops meeting these requirements, or the target maintainers no longer have interest or time, or the target shows no signs of activity and has not built for some time, or removing the target would improve the quality of the Rust codebase, we may post a PR to remove it; any such PR will be CCed to the target maintainers (and potentially other people who have previously worked on the target), to check potential interest in improving the situation.GIAt this tier, the Rust project provides no official support for a target, so we place minimal requirements on the introduction of targets.
Understood.
r? compiler-team
enable -Zrandomize-layout in debug CI builds
This builds rustc/libs/tools with `-Zrandomize-layout` on *-debug CI runners.
Only a handful of tests and asserts break with that enabled, which is promising. One test was fixable, the rest is dealt with by disabling them through new cargo features or compiletest directives.
The config.toml flag `rust.randomize-layout` defaults to false, so it has to be explicitly enabled for now.
We recently had an issue because some rustlib files were missing (like: "error[E0463]: can't find crate for rustc_ast")
when building tools that rely on rustc. This patch fixes that by copying those files as required.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
custom/external clippy support for bootstrapping
Similar to cargo, rustc, and rustfmt, this adds the support of using custom clippy on bootstrap. It’s designed for those who want to test their own clippy builds or avoid downloading the stage0 clippy.
Closes#121518
Additionally teach compiletest to ignore tests that rely on deterministic layout.
Tests themselves aren't built with randomization but they can still observe
slight changes in std or rustc
Determining this path occasionally fails locally for unknown reasons, resulting
in the build failing with an unhelpful `StripPrefixError(())` panic message.
In order to track down why that's happening, include some relevant information
in the panic message when that failure occurs.
Fix path to run clippy on rustdoc
Took me a while to find out that the path clippy expected was `src/tools/rustdoc` and not `src/librustdoc`. I think it makes more sense this way as most commands rely on source paths.
r? ```@Kobzol```
More work on `zstd` compression
r? ``@Kobzol`` as we've discussed this.
This is a draft to show the current approach of supporting zstd in compiletest, and making the tests using it unconditional.
Knowing whether llvm/lld was built with `LLVM_ENABLE_ZSTD` is quite hard, so there are two strategies. There are details in the code, and we can discuss this approach. Until we know the config used to build CI artifacts, it seems our options are somewhat limited in any case.
zlib compression seems always enabled, so we only check this in its dedicated test, allowing the test to ignore errors due to zstd not being supported.
The zstd test is made unconditional in what it tests, by relying on `needs-llvm-zstd` to be ignored when `llvm.libzstd` isn't enabled in `config.toml`.
try-job: x86_64-gnu
try-job: x86_64-msvc
try-job: x86_64-gnu-distcheck
Similar to cargo, rustc, and rustfmt, this adds the support of using custom clippy on bootstrap.
It’s designed for those who want to test their own clippy builds or avoid downloading the stage0 clippy.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
handle stage0 `cargo` and `rustc` separately
This change allows setting either `build.cargo` or `build.rustc` without requiring both to be set simultaneously, which was not possible previously.
To try it, set `build.rustc` without setting `build.cargo`, and try to bootstrap on clean build.
Blocker for https://github.com/rust-lang/rust/pull/129152
Build `library/profiler_builtins` from `ci-llvm` if appropriate
Running all of `tests/coverage` requires the LLVM profiler runtime, which requires setting `build.profiler = true`.
Historically, doing that has required checking out the entire `src/llvm-project` submodule. For compiler contributors who otherwise don't need that submodule (thanks to `download-ci-vm`), that's quite inconvenient.
However, thanks to #129116, the downloaded CI LLVM tarball now contains a copy of LLVM's `compiler-rt` directory, which includes all the files needed to build the profiler runtime. So with a little bit of extra logic in bootstrap, we can have `library/profiler_builtins` look for the `compiler-rt` files in `ci-llvm` instead of the `src/llvm-project` submodule.
This change allows setting either `build.cargo` or `build.rustc` without requiring
both to be set simultaneously, which was not possible previously.
To try it, set `build.rustc` without setting `build.cargo`, and try to bootstrap on clean build.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Rollup of 9 pull requests
Successful merges:
- #128511 (Document WebAssembly target feature expectations)
- #129243 (do not build `cargo-miri` by default on stable channel)
- #129263 (Add a missing compatibility note in the 1.80.0 release notes)
- #129276 (Stabilize feature `char_indices_offset`)
- #129350 (adapt integer comparison tests for LLVM 20 IR changes)
- #129408 (Fix handling of macro arguments within the `dropping_copy_types` lint)
- #129426 (rustdoc-search: use tighter json for names and parents)
- #129437 (Fix typo in a help diagnostic)
- #129457 (kobzol vacation)
r? `@ghost`
`@rustbot` modify labels: rollup
