Autodiff Upstreaming - rustc_codegen_ssa, rustc_middle
This PR should not be merged until the rustc_codegen_llvm part is merged.
I will also alter it a little based on what get's shaved off from the cg_llvm PR,
and address some of the feedback I received in the other PR (including cleanups).
I am putting it already up to
1) Discuss with `@jieyouxu` if there is more work needed to add tests to this and
2) Pray that there is someone reviewing who can tell me why some of my autodiff invocations get lost.
Re 1: My test require fat-lto. I also modify the compilation pipeline. So if there are any other llvm-ir tests in the same compilation unit then I will likely break them. Luckily there are two groups who currently have the same fat-lto requirement for their GPU code which I have for my autodiff code and both groups have some plans to enable support for thin-lto. Once either that work pans out, I'll copy it over for this feature. I will also work on not changing the optimization pipeline for functions not differentiated, but that will require some thoughts and engineering, so I think it would be good to be able to run the autodiff tests isolated from the rest for now. Can you guide me here please?
For context, here are some of my tests in the samples folder: https://github.com/EnzymeAD/rustbook
Re 2: This is a pretty serious issue, since it effectively prevents publishing libraries making use of autodiff: https://github.com/EnzymeAD/rust/issues/173. For some reason my dummy code persists till the end, so the code which calls autodiff, deletes the dummy, and inserts the code to compute the derivative never gets executed. To me it looks like the rustc_autodiff attribute just get's dropped, but I don't know WHY? Any help would be super appreciated, as rustc queries look a bit voodoo to me.
Tracking:
- https://github.com/rust-lang/rust/issues/124509
r? `@jieyouxu`
Clean up uses of the unstable `dwarf_version` option
- Consolidate calculation of the effective value.
- Check the target `DebuginfoKind` instead of using `is_like_msvc`.
- Add the tracking issue to the unstable book page for this feature.
cc #103057
Simplify and consolidate the way we handle construct `OutlivesEnvironment` for lexical region resolution
This is best reviewed commit-by-commit. I tried to consolidate the API for lexical region resolution *first*, then change the API when it was finally behind a single surface.
r? lcnr or reassign
Update books
## rust-lang/book
3 commits in 82a4a49789bc96db1a1b2a210b4c5ed7c9ef0c0d..fa312a343fbff01bc6cef393e326817f70719813
2025-01-22 17:14:29 UTC to 2025-01-22 15:09:26 UTC
- chore: reformat src with dprint (rust-lang/book#4211)
- Redirects: get rid of the weird gap in Ch. 20 sections! (rust-lang/book#4209)
- Document that `use` is also for `precise capturing` (rust-lang/book#4210)
## rust-lang/edition-guide
1 commits in d56e0f3a0656b7702ca466d4b191e16c28262b82..4ed5a1a4a2a7ecc2e529a5baaef04f7bc7917eda
2025-01-21 21:39:56 UTC to 2025-01-21 21:39:56 UTC
- Add alternatives for static-mut-refs (rust-lang/edition-guide#354)
## rust-lang/nomicon
3 commits in 625b200e5b33a5af35589db0bc454203a3d46d20..bc2298865544695c63454fc1f9f98a3dc22e9948
2025-01-23 19:01:24 UTC to 2025-01-20 14:37:52 UTC
- corrected grammatical error. (rust-lang/nomicon#477)
- Remove `#![start]` attribute (rust-lang/nomicon#478)
- Update guidance on uninitialized fields to use &raw mut instead of addr_of_mut! (rust-lang/nomicon#476)
## rust-lang/reference
10 commits in 293af991003772bdccf2d6b980182d84dd055942..93b921c7d3213d38d920f7f905a3bec093d2217d
2025-01-25 21:59:01 UTC to 2025-01-14 17:28:04 UTC
- distinct 'static' items never overlap (rust-lang/reference#1657)
- Change `'_static` to `'static` as an invalid lifetime parameter name (rust-lang/reference#1721)
- reword reference about inert attributes (rust-lang/reference#1719)
- Provide a better error message for broken links in mdbook-spec (rust-lang/reference#1716)
- Remove unstable vectorcall (rust-lang/reference#1717)
- Move the function pointer example (rust-lang/reference#1718)
- references and Box must be non-null (rust-lang/reference#1715)
- Fix filename for theme customization (rust-lang/reference#1711)
- Add Identifier Syntax to Several Chapters (rust-lang/reference#1597)
- move r[rules] to the left of the main body, using a grid (rust-lang/reference#1710)
Rollup of 7 pull requests
Successful merges:
- #133631 (Support QNX 7.1 with `io-sock`+libstd and QNX 8.0 (`no_std` only))
- #134358 (compiler: Set `target_abi = "ilp32e"` on all riscv32e targets)
- #135812 (Fix GDB `OsString` provider on Windows )
- #135842 (TRPL: more backward-compatible Edition changes)
- #135946 (Remove extra whitespace from rustdoc breadcrumbs for copypasting)
- #135953 (ci.py: check the return code in `run-local`)
- #136019 (Add an `unchecked_div` alias to the `Div<NonZero<_>>` impls)
r? `@ghost`
`@rustbot` modify labels: rollup
TRPL: more backward-compatible Edition changes
- Improve the discussion of `unsafe` blocks within `unsafe` functions.
- Fix formatting in Appendix A
- Incorporate line edits to Chapter 17 from NoStarch.
Support QNX 7.1 with `io-sock`+libstd and QNX 8.0 (`no_std` only)
Changes of this pull request:
1. Refactor code for qnx nto targets to share more code in file `nto_qnx.rs`
1. Add support for an additional network stack on nto qnx 7.1.
QNX 7.1 supports two network stacks:
1. `io-pkt`, which is default
2. `io-sock`, which is optional on 7.1 but default in QNX 8.0
As one can see in the [io-sock migration notes](https://www.qnx.com/developers/docs/7.1/index.html#com.qnx.doc.neutrino.io_sock/topic/migrate_app.html), this changes the libc API in a way similar to e.g. linux-gnu vs. linux-musl.
This change adds a new target which has a different value for `target_env`, so that e.g. libc can distinguish between both APIs.
2. Add initial support for QNX 8.0, thanks to AkhilTThomas. As it turned out, the problem with forking many processes still exists in QNX 8.0. Because if this, we are now using it for any QNX version (i.e. not check for `target_env` anymore).
Update emscripten std tests
This disables a bunch of emscripten tests that test things emscripten doesn't support and re-enables a whole bunch of tests which now work just fine on emscripten.
Tested with `EMCC_CFLAGS="-s MAXIMUM_MEMORY=2GB" ./x.py test library/ --target wasm32-unknown-emscripten`.
fix(libtest): Deprecate '--logfile'
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were:
(1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized.
(2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs.
Independent of the motive, regarding using or changing `--logfile`
(1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others.
(2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag
(3) The existence of `--logfile` dates back quite a ways (5cc050b265, rust-lang/rust#2127) and the history gives the
impression this more of slipped through rather than being an intended feature (see also
https://github.com/rust-lang/rust/pull/82350#discussion_r579732071). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this.
T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to
[RFC #3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
Closesrust-lang/testing-devex-team#9
Improve check-cfg expected names diagnostic
This PR improves the check-cfg `allow-same-level` test by ~~normalizing it's output and by~~ adding more context to the test.
It also filters the well known cfgs from the `expected names are` note, as to reduce the size of the diagnostic. Users can still find the full list on the [rustc book](https://doc.rust-lang.org/nightly/rustc/check-cfg.html#well-known-names-and-values), which is reinforced for Cargo users by adding a note in the Cargo check-cfg specific section.
Fixes https://github.com/rust-lang/rust/issues/135995
r? `@jieyouxu`
It seems one can't overwrite a cache entry:
```
Failed to save: Unable to reserve cache with key linkcheck--0.8.1, another job may be creating this cache. More details: Cache already exists. Scope: refs/heads/master, Key: linkcheck--0.8.1, Version: 33f8fd511bed9c91c40778bc5c27cb58425caa894ab50f9c5705d83cb78660e0
Warning: Cache save failed.
```
A proper solution is to use `restore-keys`:
https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`.
The given reasons were:
(1) Bazel can't programmatically process stdout. This seems like a
limitation in Bazel and we recommend focusing on that. If we look at
the wider Rust ecosystem, Rustc and Cargo don't support any such
mechanism and the Cargo team rejected having one. Expecting this in
libtest when its not supported elsewhere seems too specialized.
(2) Tests that leak out non-programmatic output that intermixes with
programmatic output. We acknowledge this is a problem to be evaluated
but we need to make sure we are stepping back and gathering
requirements, rather than assuming `--logfile` will fit the needs.
Independent of the motive, regarding using or changing `--logfile`
(1) Most ways to do it would be a breaking change, like if we respect
any stable `--format`. As suggested above, we could specialize this to
new `--format` values but that would be confusing for some values to
apply but not others.
(2) Other ways of solving this add new features to lib`test` when we are
instead wanting to limit the feature set it has to minimize the
compatibility surface that has to be maintained and the burden it would
put on third party harnesses which are a focus area. Examples include
`--format compact` or a `--log-format` flag
(3) The existence of `--logfile` dates back quite a ways
(5cc050b265,
rust-lang/rust#2127) and the history gives the
impression this more of slipped through rather than being an intended
feature (see also
https://github.com/rust-lang/rust/pull/82350#discussion_r579732071).
Deprecation would better match to how it has been treated.
By deprecating this, we do not expect custom test harnesses
(rust-lang/testing-devex-team#2) to implement this.
T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9
though according to
[RFC #3455](https://rust-lang.github.io/rfcs/3455-t-test.html),
this is still subject to final approval from T-libs-api.
Tidy Python improvements
Fixes display of Python formatting diffs in tidy, and refactors the code to make it simpler and more robust. Also documents Python formatting and linting in the Rustc dev guide.
Fixes: https://github.com/rust-lang/rust/issues/135942
r? `@onur-ozkan`
Implement `needs-subprocess` directive, and cleanup a bunch of tests to use `needs-{subprocess,threads}`
### Summary
Closes#128295.
- Implements `//@ needs-subprocess` directive in compiletest as requested in #128295. However, compiletest is a host tool, so we can't just try to spawn process because that spawns the process on *host*, not the *target*, under cross-compilation scenarios.
- The short-term solution is to add *Yet Another* list of allow-list targets.
- The long-term solution is to first check if a `$target` supports std, then try to run a binary to do run-time capability detection *on the target*. But that is tricky because you have to build-and-run a binary *for the target*.
- This PR picks the short-term solution, because the long-term solution is highly non-trivial, and it's already an improvement over individual `ignore-*`s all over the place.
- Opened an issue about the long-term solution in #135928.
- Documents `//@ needs-subprocess` in rustc-dev-guide.
- Replace `ignore-{wasm,wasm32,emscripten,sgx}` with `needs-{subprocess,threads}` where suitable in tests.
- Some drive-by test changes as I was trying to figure out if I could use `needs-{subprocess,threads}` and found some bits needlessly distracting.
Count of tests that use `ignore-{wasm,wasm32,emscripten,sgx}` before and after this PR:
| State | `ignore-sgx` | `ignore-wasm` | `ignore-emscripten` |
| - | - | - | - |
| Before this PR | 96 | 88 | 207 |
| After this PR | 36 | 38 | 61 |
<details>
<summary>Commands used to find out locally</summary>
```
--- before
[17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-sgx" tests | wc -l
96
[17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-wasm" tests | wc -l
88
[17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-emscripten" tests | wc -l
207
--- after
[17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-sgx" tests | wc -l
36
[17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-wasm" tests | wc -l
38
[17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-emscripten" tests | wc -l
61
```
</details>
### Review advice
- Best reviewed commit-by-commit.
- Non-trivial test changes (not mechanically simple replacements) are split into individual commits to help with review. Their individual commit messages give some basic description of the changes.
- I *could* split some test changes out into another PR, but I found that I needed to change some tests to `needs-threads`, some to `needs-subprocess`, and some needed to use *both*, so they might conflict and become very annoying.
---
r? ``@ghost`` (need to run try jobs)
try-job: x86_64-msvc-1
try-job: i686-msvc-1
try-job: i686-mingw
try-job: x86_64-mingw-1
try-job: x86_64-apple-1
try-job: aarch64-apple
try-job: aarch64-gnu
try-job: test-various
try-job: armhf-gnu
QNX SDP 8.0 comes with newly renamed QNX OS 8.0, so update the page to talk about QNX, QNX Neutrino 7.0, QNX Neutrino 7.1 or QNX OS 8.0.
Also actually add a list of target triples.
