95279f9cbd
Do not GC the current active incremental session directory when building a relatively large repo (https://github.com/oxidecomputer/omicron) on illumos under heavy CPU pressure, i saw some rustc invocations die like: ``` [..]/target/debug/incremental/<crate>-<hash>/<name>/dep-graph.part.bin: No such file or directory (os error 2) ``` a bit of debugging later and it seems that if the system is very slow, Unix-flavored `flock::Lock::new()` doesn't quite get the mutual exclusion `garbage_collect_session_directories` expects. before this patch i could reproduce this with the crate `nexus_db_queries` (in that repo) by pinning the full `cargo build` to one core and having a busy loop fighting on that same core. with this patch i cannot reproduce the issue. i took a look at how `flock::Lock` is used and i think this is the only problematic use, so i figure i'll propose this change particularly since i don't think file locking can be made.. good... for Unix in general. ------ In `setup_dep_graph`, we set up a session directory for the current incremental compilation session, load the dep graph, and then GC stale incremental compilation sessions for the crate. The freshly-created session directory ends up in this list of potentially-GC'd directories but in practice is not typically even considered for GC because the new directory is neither finalized nor `is_old_enough_to_be_collected`. Unfortunately, `is_old_enough_to_be_collected` is a simple time check, and if `load_dep_graph` is slow enough it's possible for the freshly-created session directory to be tens of seconds old already. Then, old enough to be *eligible* to GC, we try to `flock::Lock` it as proof it is not owned by anyone else, and so is a stale working directory. Because we hold the lock in the same process, the behavior of `flock::Lock` is dependent on platform-specifics about file locking APIs. `fcntl(F_SETLK)`-style locks used on non-Linux Unices do not provide mutual exclusion internal to a process. `fcntl_locking(2)` on Linux describes some relevant problems: ``` The record locks described above are associated with the process (unlike the open file description locks described below). This has some unfortunate consequences: * If a process closes any file descriptor referring to a file, then all of the process's locks on that file are released, [...] * The threads in a process share locks. In other words, a multithreaded program can't use record locking to ensure that threads don't simultaneously access the same region of a file. ``` `fcntl`-locks will appear to succeed to lock the fresh incremental compilation directory, at which point we can remove it just before using it later for incremental compilation. Saving incremental compilation state later fails and takes rustc with it with an error like ``` [..]/target/debug/incremental/crate-<hash>/<name>/dep-graph.part.bin: No such file or directory (os error 2) ``` The release-lock-on-close behavior has uncomfortable consequences for the freshly-opened file description for the lock, but I think in practice isn't an issue. If we would close the file, we failed to acquire the lock, so someone else had the lock ad we're not releasing locks prematurely. `flock(LOCK_EX)` doesn't seem to have these same issues, and because `flock::Lock::new` always opens a new file description when locking, I don't think Linux can have this issue. From reading `LockFileEx` on MSDN I *think* Windows has locking semantics similar to `flock`, but I haven't tested there at all. My conclusion is that there is no way to write a pure-POSIX `flock::Lock::new` which guarantees mutual exclusion across different file descriptions of the same file in the same process, and `flock::Lock::new` must not be used for that purpose. So, instead, avoid considering the current incremental session directory for GC in the first place. Our own `sess` is evidence we're alive and using it. |
||
|---|---|---|
| .github | ||
| compiler | ||
| library | ||
| LICENSES | ||
| src | ||
| tests | ||
| .clang-format | ||
| .editorconfig | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmodules | ||
| .ignore | ||
| .mailmap | ||
| bootstrap.example.toml | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CODE_OF_CONDUCT.md | ||
| configure | ||
| CONTRIBUTING.md | ||
| COPYRIGHT | ||
| INSTALL.md | ||
| LICENSE-APACHE | ||
| license-metadata.json | ||
| LICENSE-MIT | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| RELEASES.md | ||
| REUSE.toml | ||
| rust-bors.toml | ||
| rustfmt.toml | ||
| triagebot.toml | ||
| typos.toml | ||
| x | ||
| x.ps1 | ||
| x.py | ||
This is the main source code repository for Rust. It contains the compiler, standard library, and documentation.
Why Rust?
-
Performance: Fast and memory-efficient, suitable for critical services, embedded devices, and easily integrated with other languages.
-
Reliability: Our rich type system and ownership model ensure memory and thread safety, reducing bugs at compile-time.
-
Productivity: Comprehensive documentation, a compiler committed to providing great diagnostics, and advanced tooling including package manager and build tool (Cargo), auto-formatter (rustfmt), linter (Clippy) and editor support (rust-analyzer).
Quick Start
Read "Installation" from The Book.
Installing from Source
If you really want to install from source (though this is not recommended), see INSTALL.md.
Getting Help
See https://www.rust-lang.org/community for a list of chat platforms and forums.
Contributing
See CONTRIBUTING.md.
License
Rust is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.
See LICENSE-APACHE, LICENSE-MIT, and COPYRIGHT for details.
Trademark
The Rust Foundation owns and protects the Rust and Cargo trademarks and logos (the "Rust Trademarks").
If you want to use these names or brands, please read the Rust language trademark policy.
Third-party logos may be subject to third-party copyrights and trademarks. See Licenses for details.