rustc: Stabilize options for pipelined compilation
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in #60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).
The technical changes performed here were to stabilize the functionality
proposed in #60419 and #60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:
* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
* `diagnostic-short` - the `rendered` field of diagnostics will have a
"short" rendering matching `--error-format=short`
* `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
will be colorized with ansi color codes embedded in the string field
* `artifacts` - JSON blobs will be emitted for artifacts being emitted
by the compiler
The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.
Closes#60419Closes#60987Closes#60988
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in #60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).
The technical changes performed here were to stabilize the functionality
proposed in #60419 and #60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:
* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
* `diagnostic-short` - the `rendered` field of diagnostics will have a
"short" rendering matching `--error-format=short`
* `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
will be colorized with ansi color codes embedded in the string field
* `artifacts` - JSON blobs will be emitted for artifacts being emitted
by the compiler
The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.
Closes#60419Closes#60987Closes#60988
Now that we've fully moved to Azure Pipelines and bors has been updated
to only gate on Azure this commit removes the remaining Travis/AppVeyor
support contained in this repository. Most of the deletions here are
related to producing better output on Travis by folding certain
sections. This isn't supported by Azure so there's no need to keep it
around, and if Azure ever adds support we can always add it back!
ci: Attempt to skip a full rustc compile on dist*
Currently when we're preparing cross-compiled compilers it can take
quite some time because we have to build the compiler itself three
different times. The first is the normal bootstrap, the second is a
second build for the build platform, and the third is the actual target
architecture compiler. The second compiler was historically built
exclusively for procedural macros, and long ago we didn't actually need
it.
This commit tries out avoiding that second compiled compiler, meaning we
only compile rustc for the build platform only once. Some local testing
shows that this is promising, but bors is of course the ultimate test!
This commit furthers the previous one to ensure that we don't build an
extra stage of the compiler in CI. A test has been added to rustbuild to
ensure that this doesn't regress, and then in debugging this test it was
hunted down that the `dist::Std` target was the one erroneously pulling
in the wrong compiler.
The `dist::Std` step was updated to instead account for the "full
bootstrap" or not flag, ensuring that the correct compiler for compiling
the final standard library was used. This was another use of the
`force_use_stage1` function which was in theory supposed to be pretty
central, so existing users were all evaluated and a new function,
`Builder::compiler_for`, was introduced. All existing users of
`force_use_stage1` have been updated to use `compiler_for`, where the
semantics of `compiler_for` are similar to that of `compiler` except
that it doesn't guarantee the presence of a sysroot for the arguments
passed (as they may be modified).
Perhaps one day we can unify `compiler` and `compiler_for`, but the
usage of `Builder::compiler` is so ubiquitous it would take quite some
time to evaluate whether each one needs the sysroot or not, so it's
hoped that can be done in parallel.
Add basic CDB support to debuginfo compiletest s, to help catch `*.natvis` regressions, like those fixed in #60687.
First draft, feedback welcome.
Several Microsoft debuggers (VS, VS Code, WinDbg, CDB, ...) consume the `*.natvis` files we embed into rust `*.pdb` files. While this only tests CDB, that test coverage should help for all of them.
# Changes
## src\bootstrap
- test.rs: Run CDB debuginfo tests on MSVC targets
## src\test\debuginfo
- issue-13213.rs: CDB has trouble with this, skip for now (newly discovered regression?)
- pretty-std.rs: Was ignored, re-enable for CDB only to start with, add CDB tests.
- should-fail.rs: Add CDB tests.
## src\tools\compiletest:
- Added "-cdb" option
- Added Mode::DebugInfoCdb ("debuginfo-cdb")
- Added run_debuginfo_cdb_test[_no_opt]
- Renamed Mode::DebugInfoBoth -> DebugInfoGdbLldb ("debuginfo-gdb+lldb") since it's no longer clear what "Both" means.
- Find CDB at the default Win10 SDK install path "C:\Program Files (x86)\Windows Kits\10\Debugger\\*\cdb.exe"
- Ignore CDB tests if CDB not found.
# Issues
- `compute_stamp_hash`: not sure if there's any point in hashing `%ProgramFiles(x86)%`
- `OsString` lacks any `*.natvis` entries (would be nice to add in a followup changelist)
- DSTs (array/string slices) which work in VS & VS Code fail in CDB.
- I've avoided `Mode::DebugInfoAll` as 3 debuggers leads to pow(2,3)=8 possible combinations.
# Reference
CDB is not part of the base Visual Studio install, but can be added via the Windows 10 SDK:
https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
Installing just "Debugging Tools for Windows" is sufficient.
CDB appears to already be installed on appveyor CI, where this changelist can find it, based on it's use here:
0ffc573110/appveyor.yml (L227)
CDB commands and command line reference:
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-reference
Several Microsoft debuggers (VS, VS Code, WinDbg, CDB, ...) consume the `*.natvis` files we embed into rust `*.pdb` files.
While this only tests CDB, that test coverage should help for all of them.
CHANGES
src\bootstrap
- test.rs: Run CDB debuginfo tests on MSVC targets
src\test\debuginfo
- issue-13213.rs: CDB has trouble with this, skip for now (newly discovered regression?)
- pretty-std.rs: Was ignored, re-enable for CDB only to start with, add CDB tests.
- should-fail.rs: Add CDB tests.
src\tools\compiletest:
- Added "-cdb" option
- Added Mode::DebugInfoCdb ("debuginfo-cdb")
- Added run_debuginfo_cdb_test[_no_opt]
- Renamed Mode::DebugInfoBoth -> DebugInfoGdbLldb ("debuginfo-gdb+lldb") since it's no longer clear what "Both" means.
- Find CDB at the default Win10 SDK install path "C:\Program Files (x86)\Windows Kits\10\Debugger\*\cdb.exe"
- Ignore CDB tests if CDB not found.
ISSUES
- `compute_stamp_hash`: not sure if there's any point in hashing `%ProgramFiles(x86)%`
- `OsString` lacks any `*.natvis` entries (would be nice to add in a followup changelist)
- DSTs (array/string slices) which work in VS & VS Code fail in CDB.
- I've avoided `Mode::DebugInfoAll` as 3 debuggers leads to pow(2,3)=8 possible combinations.
REFERENCE
CDB is not part of the base Visual Studio install, but can be added via the Windows 10 SDK:
https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
Installing just "Debugging Tools for Windows" is sufficient.
CDB appears to already be installed on appveyor CI, where this changelist can find it, based on it's use here:
0ffc573110/appveyor.yml (L227)
CDB commands and command line reference:
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-reference
Allow subdirectories to be tested by x.py test
Fixes https://github.com/rust-lang/rust/issues/60718.
As far as I can tell, multiple `--test-args` flags are ignored (only the first is respected), so if you specify a subdirectory, you won't also be able to filter using `--test-args`. If you don't specify a subdirectory, `--test-args` will continue working as usual, so this is strictly an improvement on the current state of affairs.
This came out of the first Rustfix WG meeting.
One of the goals is to enable Rustfix tests for all UI tests that
trigger lints with `MachineApplicable` suggestions. In order to do that
we first want to create a tracking issue that lists all files with
missing `// run-rustfix` headers.
This PR adds a `--rustfix-coverage` flag to `./x.py` and compiletest to
list the files with the missing headers in `/tmp/rustfix_missing_coverage.txt`.
From that file we can create the tracking issue and at some point also
enforce the `// run-rustfix` flag on UI tests with `MachineApplicable`
lints.
Rework how bootstrap tools are built
This makes bootstrap tools buildable and testable in stage 0 with the downloaded bootstrap compiler, futhermore, it makes it such that they cannot be built in any other stage.
Notably, this will also mean that compiletest may need to wait a cycle before it can use changes to `libtest`, as it no longer depends on the in-tree libtest.
Introduce assembly tests suite
The change introduces a new test suite - **Assembly** tests. The motivation behind this is an ability to perform end-to-end codegen testing with LLVM backend. Turned out, NVPTX backend sometimes missing common Rust features (`i128` and libcalls in the past, and still full atomics support) due to different reasons.
Prior to this change, basic NVPTX assembly tests were implemented within `run-make` suite. Now, it's easier to write additional and maintain existing tests for the target.
cc @gnzlbg @peterhj
cc @eddyb I adjusted mangling scheme expectation, so there is no need to change the tests for #57967
compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO.
Some cross-language run-make tests need a Clang compiler that matches the LLVM version of `rustc`. Since such a compiler usually isn't available these tests (marked with the `needs-matching-clang`
directive) are ignored by default.
For some CI jobs we do need these tests to run unconditionally though. In order to support this a `--force-clang-based-tests` flag is added to compiletest. If this flag is specified, `compiletest` will fail if it can't detect an appropriate version of Clang.
@rust-lang/infra The PR doesn't yet enable the tests yet. Do you have any recommendation for which jobs to enable them?
cc #57438
r? @alexcrichton
