bors cc9bb1522e Auto merge of #83342 - Count-Count:win-console-incomplete-utf8, r=m-ou-se ... Allow writing of incomplete UTF-8 sequences to the Windows console via stdout/stderr # Problem Writes of just an incomplete UTF-8 byte sequence (e.g. `b"\xC3"` or `b"\xF0\x9F"`) to stdout/stderr with a Windows console attached error with `io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences"` even though further writes could complete the codepoint. This is currently a rare occurence since the [linewritershim](2c56ea38b0/library/std/src/io/buffered/linewritershim.rs) implementation flushes complete lines immediately and buffers up to 1024 bytes for incomplete lines. It can still happen as described in #83258. The problem will become more pronounced once the developer can switch stdout/stderr from line-buffered to block-buffered or immediate when the changes in the "Switchable buffering for Stdout" pull request (#78515) get merged. # Patch description If there is at least one valid UTF-8 codepoint all valid UTF-8 is passed through to the extracted `write_valid_utf8_to_console()` fn. The new code only comes into play if `write()` is being passed a short byte slice comprising an incomplete UTF-8 codepoint. In this case up to three bytes are buffered in the `IncompleteUtf8` struct associated with `Stdout` / `Stderr`. The bytes are accepted one at a time. As soon as an error can be detected `io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences"` is returned. Once a complete UTF-8 codepoint is received it is passed to the `write_valid_utf8_to_console()` and the buffer length is set to zero. Calling `flush()` will neither error nor write anything if an incomplete codepoint is present in the buffer. # Tests Currently there are no Windows-specific tests for console writing code at all. Writing (regression) tests for this problem is a bit challenging since unit tests and UI tests don't run in a console and suddenly popping up another console window might be surprising to developers running the testsuite and it might not work at all in CI builds. To just test the new functionality in unit tests the code would need to be refactored. Some guidance on how to proceed would be appreciated. # Public API changes * `std::str::verifications::utf8_char_width()` would be exposed as `std::str::utf8_char_width()` behind the "str_internals" feature gate. # Related issues * Fixes #83258. * PR #78515 will exacerbate the problem. # Open questions * Add tests? * Squash into one commit with better commit message?		2021-09-02 03:31:17 +00:00
..
benches	Use HTTPS links where possible	2021-06-23 16:26:46 -04:00
src	Auto merge of #83342 - Count-Count:win-console-incomplete-utf8, r=m-ou-se	2021-09-02 03:31:17 +00:00
tests	Rollup merge of #88551 - inquisitivecrystal:unsafe_cell_raw_get, r=m-ou-se	2021-09-01 09:23:31 +02:00
Cargo.toml	rfc3052: Remove authors field from Cargo manifests	2021-07-29 14:56:05 -07:00