`#[thread_local]` allows us to maintain a per-thread list of destructors. This also avoids the need to synchronize global data (which is particularly tricky within the TLS callback function).
Automatically convert paths to verbatim for filesystem operations that support it
This allows using longer paths without the user needing to `canonicalize` or manually prefix paths. If the path is already verbatim then this has no effect.
Fixes: #32689
Use BCryptGenRandom instead of RtlGenRandom on Windows.
This removes usage of RtlGenRandom on Windows, in favour of BCryptGenRandom.
BCryptGenRandom isn't available on XP, but we dropped XP support a while ago.
Fix ctrl-c causing reads of stdin to return empty on Windows.
Pressing ctrl+c (or ctrl+break) on Windows caused a blocking read of stdin to unblock and return empty, unlike other platforms which continue to block.
On ctrl-c, `ReadConsoleW` will return success, but also set `LastError` to `ERROR_OPERATION_ABORTED`.
This change detects this case, and re-tries the call to `ReadConsoleW`.
Fixes#89177. See issue for further details.
Tested on Windows 7 and Windows 10 with both MSVC and GNU toolchains
Rename `std:🧵:available_conccurrency` to `std:🧵:available_parallelism`
_Tracking issue: https://github.com/rust-lang/rust/issues/74479_
This PR renames `std:🧵:available_conccurrency` to `std:🧵:available_parallelism`.
## Rationale
The API was initially named `std:🧵:hardware_concurrency`, mirroring the [C++ API of the same name](https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency). We eventually decided to omit any reference to the word "hardware" after [this comment](https://github.com/rust-lang/rust/pull/74480#issuecomment-662045841). And so we ended up with `available_concurrency` instead.
---
For a talk I was preparing this week I was reading through ["Understanding and expressing scalable concurrency" (A. Turon, 2013)](http://aturon.github.io/academic/turon-thesis.pdf), and the following passage stood out to me (emphasis mine):
> __Concurrency is a system-structuring mechanism.__ An interactive system that deals with disparate asynchronous events is naturally structured by division into concurrent threads with disparate responsibilities. Doing so creates a better fit between problem and solution, and can also decrease the average latency of the system by preventing long-running computations from obstructing quicker ones.
> __Parallelism is a resource.__ A given machine provides a certain capacity for parallelism, i.e., a bound on the number of computations it can perform simultaneously. The goal is to maximize throughput by intelligently using this resource. For interactive systems, parallelism can decrease latency as well.
_Chapter 2.1: Concurrency is not Parallelism. Page 30._
---
_"Concurrency is a system-structuring mechanism. Parallelism is a resource."_ — It feels like this accurately captures the way we should be thinking about these APIs. What this API returns is not "the amount of concurrency available to the program" which is a property of the program, and thus even with just a single thread is effectively unbounded. But instead it returns "the amount of _parallelism_ available to the program", which is a resource hard-constrained by the machine's capacity (and can be further restricted by e.g. operating systems).
That's why I'd like to propose we rename this API from `available_concurrency` to `available_parallelism`. This still meets the criteria we previously established of not attempting to define what exactly we mean by "hardware", "threads", and other such words. Instead we only talk about "concurrency" as an abstract resource available to our program.
r? `@joshtriplett`
This allows using longer paths for filesystem operations without the user needing to `canonicalize` or manually prefix paths.
If the path is already verbatim than this has no effect.
On MinGW toolchains the various features (such as function sections)
necessary to eliminate dead function references are disabled due to
various bugs. This means that the windows sockets library will most
likely remain linked to any mingw toolchain built program that also
utilizes libstd.
That said, I made an attempt to also enable `function-sections` and
`--gc-sections` during my experiments, but the symbol references
remained, sadly.
Fix WinUWP std compilation errors due to I/O safety
I/O safety for Windows has landed in #87329. However, it does not cover UWP specific parts and prevents all UWP targets from building. See https://github.com/YtFlow/Maple/issues/18. This PR fixes these compile errors when building std for UWP targets.
Allow writing of incomplete UTF-8 sequences to the Windows console via stdout/stderr
# Problem
Writes of just an incomplete UTF-8 byte sequence (e.g. `b"\xC3"` or `b"\xF0\x9F"`) to stdout/stderr with a Windows console attached error with `io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences"` even though further writes could complete the codepoint. This is currently a rare occurence since the [linewritershim](2c56ea38b0/library/std/src/io/buffered/linewritershim.rs) implementation flushes complete lines immediately and buffers up to 1024 bytes for incomplete lines. It can still happen as described in #83258.
The problem will become more pronounced once the developer can switch stdout/stderr from line-buffered to block-buffered or immediate when the changes in the "Switchable buffering for Stdout" pull request (#78515) get merged.
# Patch description
If there is at least one valid UTF-8 codepoint all valid UTF-8 is passed through to the extracted `write_valid_utf8_to_console()` fn. The new code only comes into play if `write()` is being passed a short byte slice comprising an incomplete UTF-8 codepoint. In this case up to three bytes are buffered in the `IncompleteUtf8` struct associated with `Stdout` / `Stderr`. The bytes are accepted one at a time. As soon as an error can be detected `io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences"` is returned. Once a complete UTF-8 codepoint is received it is passed to the `write_valid_utf8_to_console()` and the buffer length is set to zero.
Calling `flush()` will neither error nor write anything if an incomplete codepoint is present in the buffer.
# Tests
Currently there are no Windows-specific tests for console writing code at all. Writing (regression) tests for this problem is a bit challenging since unit tests and UI tests don't run in a console and suddenly popping up another console window might be surprising to developers running the testsuite and it might not work at all in CI builds. To just test the new functionality in unit tests the code would need to be refactored. Some guidance on how to proceed would be appreciated.
# Public API changes
* `std::str::verifications::utf8_char_width()` would be exposed as `std::str::utf8_char_width()` behind the "str_internals" feature gate.
# Related issues
* Fixes#83258.
* PR #78515 will exacerbate the problem.
# Open questions
* Add tests?
* Squash into one commit with better commit message?
Change environment variable getters to error recoverably
This PR changes the standard library environment variable getter functions to error recoverably (i.e. not panic) when given an invalid value.
On some platforms, it is invalid for environment variable names to contain `'\0'` or `'='`, or for their values to contain `'\0'`. Currently, the standard library panics when manipulating environment variables with names or values that violate these invariants. However, this behavior doesn't make a lot of sense, at least in the case of getters. If the environment variable is missing, the standard library just returns an error value, rather than panicking. It doesn't make sense to treat the case where the variable is invalid any differently from that. See the [internals thread](https://internals.rust-lang.org/t/why-should-std-var-panic/14847) for discussion. Thus, this PR changes the functions to error recoverably in this case as well.
If desired, I could change the functions that manipulate environment variables in other ways as well. I didn't do that here because it wasn't entirely clear what to change them to. Should they error silently or do something else? If someone tells me how to change them, I'm happy to implement the changes.
This fixes#86082, an ICE that arises from the current behavior. It also adds a regression test to make sure the ICE does not occur again in the future.
`@rustbot` label +T-libs
r? `@joshtriplett`
When using `process::Command` on Windows, environment variable names must be case-preserving but case-insensitive
When using `Command` to set the environment variables, the key should be compared as uppercase Unicode but when set it should preserve the original case.
Fixes#85242
More ErrorKinds for common errnos
From the commit message of the main commit here (as revised):
```
There are a number of IO error situations which it would be very
useful for Rust code to be able to recognise without having to resort
to OS-specific code. Taking some Unix examples, `ENOTEMPTY` and
`EXDEV` have obvious recovery strategies. Recently I was surprised to
discover that `ENOSPC` came out as `ErrorKind::Other`.
Since I am familiar with Unix I reviwed the list of errno values in
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
Here, I add those that most clearly seem to be needed.
`@CraftSpider` provided information about Windows, and references, which
I have tried to take into account.
This has to be insta-stable because we can't sensibly have a different
set of ErrorKinds depending on a std feature flag.
I have *not* added these to the mapping tables for any operating
systems other than Unix and Windows. I hope that it is OK to add them
now for Unix and Windows now, and maybe add them to other OS's mapping
tables as and when someone on that OS is able to consider the
situation.
I adopted the general principle that it was usually a bad idea to map
two distinct error values to the same Rust error code. I notice that
this principle is already violated in the case of `EACCES` and
`EPERM`, which both map to `PermissionDenied`. I think this was
probably a mistake but it would be quite hard to change now, so I
don't propose to do anything about that.
However, for Windows, there are sometimes different error codes for
identical situations. Eg there are WSA* versions of some error
codes as well as ERROR_* ones. Also Windows seems to have a great
many more erorr codes. I don't know precisely what best practice
would be for Windows.
```
<strike>
```
Errno values I wasn't sure about so *haven't* included:
EMFILE ENFILE ENOBUFS ENOLCK:
These are all fairly Unix-specific resource exhaustion situations.
In practice it seemed not very likely to me that anyone would want
to handle these differently to `Other`.
ENOMEM ERANGE EDOM EOVERFLOW
Normally these don't get exposed to the Rust callers I hope. They
don't tend to come out of filesystem APIs.
EILSEQ
Hopefully Rust libraries open files in binary mode and do the
converstion in Rust. So Rust code ought not to be exposed to
EILSEQ.
EIO
The range of things that could cause this is troublesome. I found
it difficult to describe. I do think it would be useful to add this
at some point, because EIO on a filesystem operation is much more
serious than most other errors.
ENETDOWN
I wasn't sure if this was useful or, indeed, if any modern systems
use it.
ENOEXEC
It is not clear to me how a Rust program could respond to this. It
seems rather niche.
EPROTO ENETRESET ENODATA ENOMSG ENOPROTOOPT ENOSR ENOSTR ETIME
ENOTRECOVERABLE EOWNERDEAD EBADMSG EPROTONOSUPPORT EPROTOTYPE EIDRM
These are network or STREAMS related errors which I have never in
my own Unix programming found the need to do anything with. I think
someone who understands these better should be the one to try to
find good Rust names and descriptions for them.
ENOTTY ENXIO ENODEV EOPNOTSUPP ESRCH EALREADY ECANCELED ECHILD
EINPROGRESS
These are very hard to get unless you're already doing something
very Unix-specific, in which case the raw_os_error interface is
probably more suitable than relying on the Rust ErrorKind mapping.
EFAULT EBADF
These would seem to be the result of application UB.
```
</strike>
<i>(omitted errnos are discussed below, especially in https://github.com/rust-lang/rust/pull/79965#issuecomment-810468334)
Redefine `ErrorKind::Other` and stop using it in std.
This implements the idea I shared yesterday in the libs meeting when we were discussing how to handle adding new `ErrorKind`s to the standard library: This redefines `Other` to be for *user defined errors only*, and changes all uses of `Other` in the standard library to a `#[doc(hidden)]` and permanently `#[unstable]` `ErrorKind` that users can not match on. This ensures that adding `ErrorKind`s at a later point in time is not a breaking change, since the user couldn't match on these errors anyway. This way, we use the `#[non_exhaustive]` property of the enum in a more effective way.
Open questions:
- How do we check this change doesn't cause too much breakage? Will a crate run help and be enough?
- How do we ensure we don't accidentally start using `Other` again in the standard library? We don't have a `pub(not crate)` or `#[deprecated(in this crate only)]`.
cc https://github.com/rust-lang/rust/pull/79965
cc `@rust-lang/libs` `@ijackson`
r? `@dtolnay`
Use HTTPS links where possible
While looking at #86583, I wondered how many other (insecure) HTTP links were in `rustc`. This changes most other `http` links to `https`. While most of the links are in comments or documentation, there are a few other HTTP links that are used by CI that are changed to HTTPS.
Notes:
- I didn't change any to or in licences
- Some links don't support HTTPS :(
- Some `http` links were dead, in those cases I upgraded them to their new places (all of which used HTTPS)
Dump mingw-64's error codes into our source tree.
I have verified with these runes:
$ f=library/std/src/sys/windows/c/errors.rs
$ diff -ub <(git-cat-file blob HEAD~:$f | sort) <(cat $f | perl -pe 's/WSABASEERR \+ (\d+)/10000 + $1/e' |sort) |grep ^- |less
that this does not change any existing values.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
We're going to add many more of these.
This commit is pure code motion, plus the necessary administrivia, as
I have veried with the following runes:
$ git-diff HEAD~ | grep '^+' |sort >plus
$ git-diff HEAD~ | grep '^-' | perl -pe 's/^-/+/' |sort >min
$ diff -ub min plus |less
The output is precisely the expected `mod` and `use` directives.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
DWORD is a type alias for u32, so this makes no difference.
But this entry is anomalous and in my forthcoming commits I am going
to import many errors wholesale, and I spotted that my wholesale
import didn't match what was here.
CC: Chris Denton <christophersdenton@gmail.com>
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
