user0/rust
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Auto merge of #88362 - pietroalbini:bump-stage0, r=Mark-Simulacrum

Browse source 
Pin bootstrap checksums and add a tool to update it automatically

⚠️ ⚠️ This is just a proactive hardening we're performing on the build system, and it's not prompted by any known compromise. If you're aware of security issues being exploited please [check out our responsible disclosure page](https://www.rust-lang.org/policies/security). ⚠️ ⚠️

---

This PR aims to improve Rust's supply chain security by pinning the checksums of the bootstrap compiler downloaded by `x.py`, preventing a compromised `static.rust-lang.org` from affecting building the compiler. The checksums are stored in `src/stage0.json`, which replaces `src/stage0.txt`. This PR also adds a tool to automatically update the bootstrap compiler.

The changes in this PR were originally discussed in [Zulip](https://zulip-archive.rust-lang.org/stream/241545-t-release/topic/pinning.20stage0.20hashes.html).

## Potential attack

Before this PR, an attacker who wanted to compromise the bootstrap compiler would "just" need to:

1. Gain write access to `static.rust-lang.org`, either by compromising DNS or the underlying storage.
2. Upload compromised binaries and corresponding `.sha256` files to `static.rust-lang.org`.

There is no signature verification in `x.py` as we don't want the build system to depend on GPG. Also, since the checksums were not pinned inside the repository, they were downloaded from `static.rust-lang.org` too: this only protected from accidental changes in `static.rust-lang.org` that didn't change the `*.sha256` files. The attack would allow the attacker to compromise past and future invocations of `x.py`.

## Mitigations introduced in this PR

This PR adds pinned checksums for all the bootstrap components in `src/stage0.json` instead of downloading the checksums from `static.rust-lang.org`. This changes the attack scenario to:

1. Gain write access to `static.rust-lang.org`, either by compromising DNS or the underlying storage.
2. Upload compromised binaries to `static.rust-lang.org`.
3. Land a (reviewed) change in the `rust-lang/rust` repository changing the pinned hashes.

Even with a successful attack, existing clones of the Rust repository won't be affected, and once the attack is detected reverting the pinned hashes changes should be enough to be protected from the attack. This also enables further mitigations to be implemented in following PRs, such as verifying signatures when pinning new checksums (removing the trust on first use aspect of this PR) and adding a check in CI making sure a PR updating the checksum has not been tampered with (see the future improvements section).

## Additional changes

There are additional changes implemented in this PR to enable the mitigation:

* The `src/stage0.txt` file has been replaced with `src/stage0.json`. The reasoning for the change is that there is existing tooling to read and manipulate JSON files compared to the custom format we were using before, and the slight challenge of manually editing JSON files (no comments, no trailing commas) are not a problem thanks to the new `bump-stage0`.

* A new tool has been added to the repository, `bump-stage0`. When invoked, the tool automatically calculates which release should be used as the bootstrap compiler given the current version and channel, gathers all the relevant checksums and updates `src/stage0.json`. The tool can be invoked by running:

  ```
  ./x.py run src/tools/bump-stage0
  ```

* Support for downloading releases from `https://dev-static.rust-lang.org` has been removed, as it's not possible to verify checksums there (it's customary to replace existing artifacts there if a rebuild is warranted). This will require a change to the release process to avoid bumping the bootstrap compiler on beta before the stable release.

## Future improvements

* Add signature verification as part of `bump-stage0`, which would require the attacker to also obtain the release signing keys in order to successfully compromise the bootstrap compiler. This would be fine to add now, as the burden of installing the tool to verify signatures would only be placed on whoever updates the bootstrap compiler, instead of everyone compiling Rust.

* Add a check on CI that ensures the checksums in `src/stage0.json` are the expected ones. If a PR changes the stage0 file CI should also run the `bump-stage0` tool and fail if the output in CI doesn't match the committed file. This prevents the PR author from tweaking the output of the tool manually, which would otherwise be close to impossible for a human to detect.

* Automate creating the PRs bumping the bootstrap compiler, by setting up a scheduled job in GitHub Actions that runs the tool and opens a PR.

* Investigate whether a similar mitigation can be done for "download from CI" components like the prebuilt LLVM.

r? `@Mark-Simulacrum`
This commit is contained in:
bors 2021-09-06 16:01:17 +00:00
commit 8ceea01bb4
13 changed files with 677 additions and 149 deletions
Download patch file Download diff file Expand all files Collapse all files

14
src/tools/bump-stage0/Cargo.toml Normal file
View file

@ -0,0 +1,14 @@
[package]
name = "bump-stage0"
version = "0.1.0"
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
anyhow = "1.0.34"
curl = "0.4.38"
indexmap = { version = "1.7.0", features = ["serde"] }
serde = { version = "1.0.125", features = ["derive"] }
serde_json = "1.0.59"
toml = "0.5.7"

204
src/tools/bump-stage0/src/main.rs Normal file
View file

@ -0,0 +1,204 @@
use anyhow::Error;
use curl::easy::Easy;
use indexmap::IndexMap;
use std::collections::HashMap;
use std::convert::TryInto;
const DIST_SERVER: &str = "https://static.rust-lang.org";
const COMPILER_COMPONENTS: &[&str] = &["rustc", "rust-std", "cargo"];
const RUSTFMT_COMPONENTS: &[&str] = &["rustfmt-preview"];
struct Tool {
channel: Channel,
version: [u16; 3],
checksums: IndexMap<String, String>,
}
impl Tool {
fn new() -> Result<Self, Error> {
let channel = match std::fs::read_to_string("src/ci/channel")?.trim() {
"stable" => Channel::Stable,
"beta" => Channel::Beta,
"nightly" => Channel::Nightly,
other => anyhow::bail!("unsupported channel: {}", other),
};
// Split "1.42.0" into [1, 42, 0]
let version = std::fs::read_to_string("src/version")?
.trim()
.split('.')
.map(|val| val.parse())
.collect::<Result<Vec<_>, _>>()?
.try_into()
.map_err(|_| anyhow::anyhow!("failed to parse version"))?;
Ok(Self { channel, version, checksums: IndexMap::new() })
}
fn update_json(mut self) -> Result<(), Error> {
std::fs::write(
"src/stage0.json",
format!(
"{}\n",
serde_json::to_string_pretty(&Stage0 {
comment: "Generated by `./x.py run src/tools/bump-stage0`. \
Run that command again to update the bootstrap compiler.",
dist_server: DIST_SERVER.into(),
compiler: self.detect_compiler()?,
rustfmt: self.detect_rustfmt()?,
checksums_sha256: {
// Keys are sorted here instead of beforehand because values in this map
// are added while filling the other struct fields just above this block.
self.checksums.sort_keys();
self.checksums
}
})?
),
)?;
Ok(())
}
// Currently Rust always bootstraps from the previous stable release, and in our train model
// this means that the master branch bootstraps from beta, beta bootstraps from current stable,
// and stable bootstraps from the previous stable release.
//
// On the master branch the compiler version is configured to `beta` whereas if you're looking
// at the beta or stable channel you'll likely see `1.x.0` as the version, with the previous
// release's version number.
fn detect_compiler(&mut self) -> Result<Stage0Toolchain, Error> {
let channel = match self.channel {
Channel::Stable | Channel::Beta => {
// The 1.XX manifest points to the latest point release of that minor release.
format!("{}.{}", self.version[0], self.version[1] - 1)
}
Channel::Nightly => "beta".to_string(),
};
let manifest = fetch_manifest(&channel)?;
self.collect_checksums(&manifest, COMPILER_COMPONENTS)?;
Ok(Stage0Toolchain {
date: manifest.date,
version: if self.channel == Channel::Nightly {
"beta".to_string()
} else {
// The version field is like "1.42.0 (abcdef1234 1970-01-01)"
manifest.pkg["rust"]
.version
.split_once(' ')
.expect("invalid version field")
.0
.to_string()
},
})
}
/// We use a nightly rustfmt to format the source because it solves some bootstrapping issues
/// with use of new syntax in this repo. For the beta/stable channels rustfmt is not provided,
/// as we don't want to depend on rustfmt from nightly there.
fn detect_rustfmt(&mut self) -> Result<Option<Stage0Toolchain>, Error> {
if self.channel != Channel::Nightly {
return Ok(None);
}
let manifest = fetch_manifest("nightly")?;
self.collect_checksums(&manifest, RUSTFMT_COMPONENTS)?;
Ok(Some(Stage0Toolchain { date: manifest.date, version: "nightly".into() }))
}
fn collect_checksums(&mut self, manifest: &Manifest, components: &[&str]) -> Result<(), Error> {
let prefix = format!("{}/", DIST_SERVER);
for component in components {
let pkg = manifest
.pkg
.get(*component)
.ok_or_else(|| anyhow::anyhow!("missing component from manifest: {}", component))?;
for target in pkg.target.values() {
for pair in &[(&target.url, &target.hash), (&target.xz_url, &target.xz_hash)] {
if let (Some(url), Some(sha256)) = pair {
let url = url
.strip_prefix(&prefix)
.ok_or_else(|| {
anyhow::anyhow!("url doesn't start with dist server base: {}", url)
})?
.to_string();
self.checksums.insert(url, sha256.clone());
}
}
}
}
Ok(())
}
}
fn main() -> Result<(), Error> {
let tool = Tool::new()?;
tool.update_json()?;
Ok(())
}
fn fetch_manifest(channel: &str) -> Result<Manifest, Error> {
Ok(toml::from_slice(&http_get(&format!(
"{}/dist/channel-rust-{}.toml",
DIST_SERVER, channel
))?)?)
}
fn http_get(url: &str) -> Result<Vec<u8>, Error> {
let mut data = Vec::new();
let mut handle = Easy::new();
handle.fail_on_error(true)?;
handle.url(url)?;
{
let mut transfer = handle.transfer();
transfer.write_function(|new_data| {
data.extend_from_slice(new_data);
Ok(new_data.len())
})?;
transfer.perform()?;
}
Ok(data)
}
#[derive(Debug, PartialEq, Eq)]
enum Channel {
Stable,
Beta,
Nightly,
}
#[derive(Debug, serde::Serialize)]
struct Stage0 {
#[serde(rename = "__comment")]
comment: &'static str,
dist_server: String,
compiler: Stage0Toolchain,
rustfmt: Option<Stage0Toolchain>,
checksums_sha256: IndexMap<String, String>,
}
#[derive(Debug, serde::Serialize)]
struct Stage0Toolchain {
date: String,
version: String,
}
#[derive(Debug, serde::Deserialize)]
struct Manifest {
date: String,
pkg: HashMap<String, ManifestPackage>,
}
#[derive(Debug, serde::Deserialize)]
struct ManifestPackage {
version: String,
target: HashMap<String, ManifestTargetPackage>,
}
#[derive(Debug, serde::Deserialize)]
struct ManifestTargetPackage {
available: bool,
url: Option<String>,
hash: Option<String>,
xz_url: Option<String>,
xz_hash: Option<String>,
}