Enforce that PR CI jobs are a subset of Auto CI jobs modulo carve-outs
### Background
Currently, it is possible for a PR with red PR-only CI to pass Auto CI, then all subsequent PR CI runs will be red until that is fixed, even in completely unrelated PRs. For instance, this happened with PR-CI-only Spellcheck (rust-lang/rust#144183).
See more discussions at [#t-infra > Spellcheck workflow now fails on all PRs (tree bad?)](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/Spellcheck.20workflow.20now.20fails.20on.20all.20PRs.20.28tree.20bad.3F.29/with/529769404).
### CI invariant: PR CI jobs are a subset of Auto CI jobs modulo carve-outs
To prevent red PR CI in completely unrelated subsequent PRs and PR CI runs, we need to maintain an invariant that **PR CI jobs are a subset of Auto CI jobs modulo carve-outs**.
This is **not** a "strict" subset relationship: some jobs necessarily have to differ under PR CI and Auto CI environments, at least in the current setup. Still, we can try to enforce a weaker "subset modulo carve-outs" relationship between CI jobs and their corresponding Auto jobs. For instance:
- `x86_64-gnu-tools` will have `auto`-only env vars like `DEPLOY_TOOLSTATES_JSON: toolstates-linux.json`.
- `tidy` will want to `continue_on_error: true` in PR CI to allow for more "useful" compilation errors to also be reported, whereas it should be `continue_on_error: false` in Auto CI to prevent wasting Auto CI resources.
The **carve-outs** are:
1. `env` variables.
2. `continue_on_error`.
We enforce this invariant through `citool`, so only affects job definitions that are handled by `citool`. Notably, this is not sufficient *alone* to address the CI-only Spellcheck issue (rust-lang/rust#144183). To carry out this enforcement, we modify `citool` to auto-register PR jobs as Auto jobs with `continue_on_error` overridden to `false` **unless** there's an overriding Auto job for the PR job of the same name that only differs by the permitted **carve-outs**.
### Addressing the Spellcheck PR-only CI issue
Note that Spellcheck currently does not go through `citool` or `bootstrap`, and is its own GitHub Actions workflow. To actually address the PR-CI-only Spellcheck issue (rust-lang/rust#144183), and carry out the subset-modulo-carve-outs enforcement universally, this PR additionally **removes the current Spellcheck implementation** (a separate GitHub Actions Workflow). That is incompatible with Homu unless we do some hacks in the main CI workflow.
This effectively partially reverts rust-lang/rust#134006 (the separate workflow part, not the tidy extra checks component), but is not prejudice against relanding the `typos`-based spellcheck in another implementation that goes through the usual bootstrap CI workflow so that it does work with Homu. The `typos`-based spellcheck seems to have a good false-positive rate.
Closesrust-lang/rust#144183.
---
r? infra-ci
bootstrap: add package.json and package-lock.json to dist tarball
this ensures that js-related tests can still be run from within such a dist tarball.
followup to rust-lang/rust#142924
r? ```````@Kobzol```````
coverage: Enlarge empty spans during MIR instrumentation, not codegen
This re-lands the part of rust-lang/rust#140847 that was (hopefully) not responsible for the coverage-instrumentation regressions that caused that PR to be reverted.
---
Enlarging empty spans was historically performed during MIR instrumentation, but had to be moved to codegen as part of larger changes in rust-lang/rust#134497, leading to the status quo. But now there should be no reason not to move that step back to its more logical home in instrumentaion.
mbe: Use concrete type for `get_unused_rule`
Rather than adding `get_unused_rule` to the `TTMacroExpander` trait, put
it on the concrete `MacroRulesMacroExpander`, and downcast to that type
via `Any` in order to call it.
Suggested-by: Vadim Petrochenkov <vadim.petrochenkov@gmail.com>
r? ```````@petrochenkov```````
Clean `rustc/parse/src/lexer` to improve maintainability
This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value
In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.
In the third commit, I moves `eof_err` function below parsing logic for better code flow.
In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.
r? compiler
Fix broken TLS destructors on 32-bit win7
Fixesrust-lang/rust#141300
On the 32-bit win7 target, we use OS TLS instead of native TLS, due to issues with how the OS handles alignment. Unfortunately, this caused issues due to the TLS destructors not running, causing memory leaks among other problems.
On Windows, to support OS TLS, the TlsAlloc family of function is used by Rust. This function does not support TLS destructors at all. However, rust has some code to emulate those destructors, by leveraging the TLS support functionality found in the MSVC CRT (specifically, in tlssup.c of the CRT).
To use this functionality, the user must do two things:
1. They must put the address to their callback in a section between `.CRT$XLB` and `.CRT$XLY`.
2. They must add a reference to `_tls_used` (or `__tls_used` on x86) to make sure the TLS support code in tlssup.c isn't garbage collected by the linker.
Prior to this commit, this second bit wasn't being done properly by the Rust TLS support code. Instead of adding a reference to _tls_used, it instead had a reference to its own callback to prevent it from getting GC'd by the linker. While this is _also_ necessary, not having a reference on _tls_used made the entire support non-functional.
This commit reworks the code to:
1. Add an unconditional `#[used]` attribute on the CALLBACK, which should be enough to prevent it from getting GC'd by the linker.
2. Add a reference to `_tls_used`, which should pull the TLS support code into the Rust programs and not let it be GC'd by the linker.
Move `std_detect` into stdlib
This PR moves the `std_detect` crate from `stdarch` to be a part of rust-lang/rust instead.
The first commit actually moves the whole directory from the stdarch Josh subtree, so that git blame history is kept intact. Then I had to make a few changes to appease `tidy`.
The most complex thing here is porting the tests. We can't have `std_detect` both in r-l/r and stdarch, because they could get desynchronized, so we have to perform the move more or less "atomically", which means that we also have to port all the existing `std_detect` tests from the `stdarch` repository.
The stdarch repo runs the following `std_detect` tests:
### Build
The `build-std-detect.sh` script (e2b6512aed/ci/build-std-detect.sh) builds `std_detect` using the nightly compiler for several targets. This will be subsumed by normal `x build library` on our Tier 1/2 targets. However, the stdarch repository also tests the following targets:
- aarch64-unknown-freebsd
- armv6-unknown-freebsd
- powerpc-unknown-freebsd
- powerpc64-unknown-freebsd
- aarch64-unknown-openbsd
Which we don't build/test on our CI currently. I think we have mostly two options here:
1) Ignore these targets
2) Create a special CI job that will build stage 1 rustc and then cross-compile std (or just the `std_detect` crate?) for these targets.
### Documentation
The `dox.sh` script (3fec5adcd5/ci/dox.sh) builds and documents `std_detect` for several targets. All of them are Tier 2/we have `dist-` jobs for them, so I think that we can just skip this and let our normal CI subsume it?
### Tests
The `run.sh` script (1b201cec2c/ci/run.sh) runs `cargo test` on `std_detect` with a bunch of variations of feature flags. This will be subsumed by `x test library` in our CI. The only problem is that `stdarch` runs these tests for a ludicrous number of targets:
```
- tuple: i686-unknown-linux-gnu
- tuple: x86_64-unknown-linux-gnu
- tuple: arm-unknown-linux-gnueabihf
- tuple: armv7-unknown-linux-gnueabihf
- tuple: aarch64-unknown-linux-gnu
- tuple: aarch64_be-unknown-linux-gnu
- tuple: riscv32gc-unknown-linux-gnu
- tuple: riscv64gc-unknown-linux-gnu
- tuple: powerpc-unknown-linux-gnu
- tuple: powerpc64-unknown-linux-gnu
- tuple: powerpc64le-unknown-linux-gnu
- tuple: s390x-unknown-linux-gnu
- tuple: i586-unknown-linux-gnu
- tuple: nvptx64-nvidia-cuda
- tuple: thumbv6m-none-eabi
- tuple: thumbv7m-none-eabi
- tuple: thumbv7em-none-eabi
- tuple: thumbv7em-none-eabihf
- tuple: loongarch64-unknown-linux-gnu
- tuple: wasm32-wasip1
- tuple: x86_64-apple-darwin
- tuple: x86_64-apple-ios-macabi
- tuple: aarch64-apple-darwin
- tuple: aarch64-apple-ios-macabi
- tuple: x86_64-pc-windows-msvc
- tuple: i686-pc-windows-msvc
- tuple: aarch64-pc-windows-msvc
- tuple: x86_64-pc-windows-gnu
- tuple: aarch64-unknown-linux-gnu
- tuple: aarch64_be-unknown-linux-gnu
- tuple: armv7-unknown-linux-gnueabihf
- tuple: loongarch64-unknown-linux-gnu
- tuple: powerpc-unknown-linux-gnu
- tuple: powerpc64-unknown-linux-gnu
- tuple: powerpc64le-unknown-linux-gnu
- tuple: riscv32gc-unknown-linux-gnu
- tuple: riscv64gc-unknown-linux-gnu
- tuple: s390x-unknown-linux-gnu
- tuple: x86_64-unknown-linux-gnu
- tuple: aarch64-apple-darwin
- tuple: aarch64-apple-ios-macabi
```
We definitely do not run *tests* for all of these targets on our CI.
# Outcome
We have decided to just subsume std_detect tests by our normal test suite for now, and not create a separate CI job. Therefore, this PR performs the following changes in target testing for `std_detect`:
The following T3 targets would go from "build" to "nothing":
```
aarch64-unknown-freebsd (T3)
armv6-unknown-freebsd (T3)
powerpc-unknown-freebsd (T3)
powerpc64-unknown-freebsd (T3)
aarch64-unknown-openbsd (T3)
```
The following T3 targets would go from "test" to "nothing":
```
aarch64_be-unknown-linux-gnu (T3)
riscv32gc-unknown-liux-gnu (T3)
```
The following T2 targets would go from "test" to "build":
```
arm-unknown-linux-gnueabihf (T2)
armv7-unknown-linux-gnueabihf (T2)
riscv64gc-unknown-linux-gnu (T2)
powerpc-unknown-linux-gnu (T2)
powerpc64-unknown-linux-gnu (T2)
powerpc64le-unknown-linux-gnu (T2)
s390x-unknown-linux-gnu (T2)
i586-unknown-linux-gnu (T2)
loongarch64-unknown-linux-gnu (T2)
wasm32-wasip1 (T2)
x86_64-apple-ios-macabi (T2)
aarch64-apple-ios-macabi (T2)
aarch64-pc-windows-msvc (T2)
armv7-unknown-linux-gnueabihf (T2)
loongarch64-unknown-linux-gnu (T2)
powerpc-unknown-linux-gnu (T2)
```
I have confirmed in https://github.com/rust-lang/stdarch/pull/1873 that the current version of this PR would pass stdarch's CI testsuite.
r? `@ghost`
try-job: armhf-gnu
try-job: arm-android
Implement AST visitors using a derive macro.
AST visitors are large and error-prone beasts. This PR attempts to write them using a derive macro.
The design uses three traits: `Visitor`, `Visitable`, `Walkable`.
- `Visitor` is the trait implemented by downstream crates, it lists `visit_stuff` methods, which call `Walkable::walk_ref` by default;
- `Walkable` is derived using the macro, the generated `walk_ref` method calls `Visitable::visit` on each component;
- `Visitable` is implemented by `common_visitor_and_walkers` macro, to call the proper `Visitor::visit_stuff` method if it exists, to call `Walkable::walk_ref` if there is none.
I agree this is quite a lot of spaghetti macros. I'm open to suggestions on how to reduce the amount of boilerplate code.
If this PR is accepted, I believe the same design can be used for the HIR visitor.
Rename `tests/{assembly,codegen}` into `tests/{assembly,codegen}-llvm` and ignore these testsuites if configured backend doesn't match
Follow-up of https://github.com/rust-lang/rust/pull/144125.
This PR changes `compiletest` so that `asm` tests are only run if they match the current codegen backend. To better reflect it, I renamed the `tests/ui/asm` folder into `tests/ui/asm-llvm`. Like that, we can add new asm tests for other backends if we want without needing to add extra code to `compiletest`.
Next step will be to use the new code annotations added in rust-lang/rust#144125 to ignore ui tests failing in cg_gcc until it's fixed on our side.
cc `@antoyo` `@oli-obk`
r? `@Kobzol`
Rollup of 7 pull requests
Successful merges:
- rust-lang/rust#142454 (Add modern AVR mcus like avr128db28 and attiny3224)
- rust-lang/rust#142924 (tidy: move rustdoc js stuff into a tidy extra check)
- rust-lang/rust#143373 (Unquerify maybe_unused_trait_imports.)
- rust-lang/rust#144082 (tests: cover more `exported_private_dependencies` cases)
- rust-lang/rust#144126 (Fix empty target_config in apply_rust_config bootstrap)
- rust-lang/rust#144164 ( opt-dist: add an option for setting path to stage0 root)
- rust-lang/rust#144265 (Dont ICE on copy error being suppressed due to overflow)
r? `@ghost`
`@rustbot` modify labels: rollup
flt2dec: replace for loop by iter_mut
Perf is explored in https://github.com/rust-lang/rust/issues/144118, which initially showed small losses, but then also showed significant gains. Both are real, but given the smallness of the losses, this seems a good change.
opt-dist: add an option for setting path to stage0 root
in MSYS2 we have problems with stage0 for *-gnullvm hosts because prebuilt dist tarballs will be
available starting from 1.90.0-beta. also this change helps to match bootstrap.toml config
r? Kobzol
try-job: dist-x86_64-msvc
try-job: dist-x86_64-linux
Fix empty target_config in apply_rust_config bootstrap
This PR fixes the issue of an empty target_config in apply_rust_config, which was caused by the ordering of TOML config parsing. This was inadvertently introduced during the last config refactor. The test and the corresponding configuration order have been corrected in this PR.
r? ```@Kobzol```
tests: cover more `exported_private_dependencies` cases
This PR adds tests for all missing cases from rust-lang/rust#71043 and some on top of that. I believe with this, that issue can be closed.
Some of the lints can be improved, e.g. `provided_impl_trait` and `impl From<PublicWithStdImpl> for OtherType` lint twice.
cc ```@epage``` in case you want to double check I didn't miss anything.
