Currently mirrors are stored in the rust-lang-ci2 S3 bucket along with
CI toolchains. This is problematic for multiple reasons:
- CI IAM credentials are allowed to both edit and delete those files.
A malicious user gaining access to those credentials would be able to
change our mirrored dependencies, possibly backdooring the compiler.
- Contents of the rust-lang-ci2 bucket are disposable except for the
mirrors' content. When we implement backups for S3 buckets we'd have
to replicate just that part of the bucket, complicating the backup
logic and increasing the chance of mistakes. A standalone bucket will
be way easier to backup.
This commit switches our CI to use the new rust-lang-ci-mirrors bucket.
Added support for armv7-unknown-linux-gnueabi/musleabi
Fixes#63101
Some things that are not done and I hope someone can help me with:
* During the ci build of `armv7-unknown-linux-gnueabi` `openssl` must be built (to build cargo) but `openssl` does not yet support this target. This feels slightly like a chicken-and-egg problem, any feedback is welcome.
* Should I add any tests for any of these targets?
ci: fix toolstate not pushing data for Linux
A recent commit modified toolstate to only push updated data when the `TOOLSTATE_PUBLISH` environment variable is present. This worked fine on Windows but failed on Linux, since Linux jobs run inside Docker containers and the variable wasn't forwarded inside it.
This changes the Docker startup code to set the `TOOLSTATE_PUBLISH` enviornment variable inside the container if it's present outside.
r? @alexcrichton
fixes https://github.com/rust-lang/rust/issues/63190
A recent commit modified toolstate to only push updated data when the
TOOLSTATE_PUBLISH environment variable is present. This worked fine on
Windows but failed on Linux, since Linux jobs run inside Docker
containers and the variable wasn't forwarded inside it.
This changes the Docker startup code to set the TOOLSTATE_PUBLISH
enviornment variable inside the container if it's present outside.
ci: gate toolstate repo pushes on the TOOLSTATE_PUBLISH envvar
This PR fixes toolstate failing to push on the LinuxTools PR builder by gating the pushes on the new `TOOLSTATE_PUBLISH` environment variable, which is set on prod credentials but not on the PR ones. The old code checked whether the access token was set, but that doesn't work due to an Azure quirk.
For a bit of background, secret environment variables are not available by default, but each step needs to explicitly declare which secret vars to load:
```yaml
- bash: echo foo
env:
SECRET_VAR: $(SECRET_VAR)
```
This works fine when the variable is present but when it's missing, instead of setting `SECRET_VAR` to an empty string or just not setting it at all, Azure Pipelines puts the literal `$(SECRET_VAR)` as the content, which completly breaks the old check we had. I tried almost every thing to make this work in a sensible way, and the only conclusion I reached is to set the variable at the top level with the runtime expression evaluation syntax, which sets the variable to an empty string if missing:
```yaml
# At the top:
variables:
- name: MAYBE_SECRET_VAR
value: $[ variables.MAYBE_SECRET_VAR ]
# In the step:
- bash: echo foo
env:
SECRET_VAR: $(MAYBE_SECRET_VAR)
```
While that *could've worked* it was ugly and messy, so I just opted to add yet another non-secret variable.
r? @alexcrichton
fixes#62811
Unfortunately due to an Azure quirk the TOOLSTATE_REPO_ACCESS_TOKEN is
not suitable to gate whether to push new commits to the repo, as if it's
not defined on the Azure side it will actually be set to the literal
`$(TOOLSTATE_REPO_ACCESS_TOKEN)`, which screws everything up.
This instead adds another, non-secret environment variable to gate
publishing: TOOLSTATE_PUBLISH. As non-secret environment variables
behave correctly this fixes the issue.
This commit updates some of our assorted Azure/CI configuration to
prepare for some 4-core machines coming online. We're still in the
process of performance testing them to get final numbers, but some
changes are worth landing ahead of this. The updates here are:
* Use `C:/` instead of `D:/` for submodule checkout since it should have
plenty of space and the 4-core machines won't have `D:/`
* Update `lzma-sys` to 0.1.14 which has support for VS2019, where 0.1.10
doesn't.
* Update `src/ci/docker/run.sh` to work when it itself is running inside
of a docker container (see the comment in the file for more info)
* Print step timings on the `try` branch in addition to the `auto`
branch in. The logs there should be seen by similarly many humans (not
many) and can be useful for performance analysis after a `try` build
runs.
* Install the WIX and InnoSetup tools manually on Windows instead of
relying on pre-installed copies on the VM. This gives us more control
over what's being used on the Azure cloud right now (we control the
version) and in the 4-core machines these won't be pre-installed. Note
that on AppVeyor we actually already were installing InnoSetup, we
just didn't carry that over on Azure!
Update to LLVM 9 trunk
Following the preparatory changes in #62474, this updates the LLVM submodule to https://github.com/rust-lang/llvm-project/tree/rustc/9.0-2019-07-12 and:
* Changes the LLVM Rust bindings to account for the new SubtargetSubTypeKV.
* Adjusts a codegen test for the new form of the byval attribute that takes a type.
* Makes a PGO codegen test more liberal with regard to order and linkage.
* Builds InstrProfilingPlatformWindows.c as part of libprofiler_builtins.
* Moves registration of additional passes (in particular sanitizers) to the end of the module pass manager.
* Disables LLDB on builders.
r? @alexcrichton
Now that we've fully moved to Azure Pipelines and bors has been updated
to only gate on Azure this commit removes the remaining Travis/AppVeyor
support contained in this repository. Most of the deletions here are
related to producing better output on Travis by folding certain
sections. This isn't supported by Azure so there's no need to keep it
around, and if Azure ever adds support we can always add it back!
Update musl to 1.1.22
This is the latest available version. I noticed Rust wasn't using the
latest version when I attempted to have Cargo link object files produced
outside of Cargo / Rust's toolchain and was getting missing symbol
errors. Those missing symbols were added in 1.1.21 and 1.1.22.
I'm not fully sure of the implications of this change or how
comprehensive the test coverage is. Upstream changes in 1.1.21 and
1.1.22 can be found at
https://git.musl-libc.org/cgit/musl/tree/WHATSNEW?h=v1.1.22&id=e97681d6f2c44bf5fa9ecdd30607cb63c780062e#n1989.
The PR builder on Azure currently takes 2.5h which is a bit long, so
this commit disables debug assertions and llvm assertions in an attempt
to speed up that builder and have PR builds come back a bit more
quickly. Other builders continue to enable debug assertions and test the
compiler there.
Limit dylib symbols
This makes `windows-gnu` match the behavior of `windows-msvc`. It probably doesn't make sense to export these symbols on other platforms either.
Currently just run it through its paces but don't actually push to
official locations. Instead let's just push to a separate fork (mine) as
well as open issues in a separate fork (mine). Make sure that people
aren't pinged for these issues as well!
This should hopefully ensure that everything is working on Azure and
give us a chance to work through any issues that come up.
ci: Disable LLVM/debug assertions for distcheck
The purpose of distcheck is to test `./x.py test` from a tarball, not to
test that all assertions pass all the time. These assertions are largely
just redundant with other builders, so skip the assertions for now and
save a good chunk of time on CI.
cc #61185
checktools: unify grepping the TOOLSTATE file
The file was grepped twice but in a different way. This unifies the code to make sure it is consistent. Or were these deliberately not doing the same thing? That seems strange though.
I wouldn't know how to test these changes.
The purpose of distcheck is to test `./x.py test` from a tarball, not to
test that all assertions pass all the time. These assertions are largely
just redundant with other builders, so skip the assertions for now and
save a good chunk of time on CI.
cc #61185
This shaves of 50 minutes of cycle time on Azure and will likely also
save a significant chunk of time on Travis. The assertions here aren't
really buying us much over other builders with assertions already
enabled, so let's disable them for this builder.
cc #61185
Update musl-cross-make to 0.9.8
This version uses musl 1.1.22 and GCC 6.4.0 by default. It also
contains support for newer binutils and GCC versions, should we
want to bump those as well. But I'm purposefully limiting this
patch in order to reduce the surface area for controversy.
ci: Favor SCRIPT instead of RUST_CHECK_TARGET
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
