user0/rust
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
my rust fork https://www.rust-lang.org
268238 commits 10 branches 154 tags 1.3 GiB
Find a file
bors 36780360b6 Auto merge of #125679 - clarfonthey:escape_ascii, r=joboet 
Optimize `escape_ascii` using a lookup table

Based upon my suggestion here: https://github.com/rust-lang/rust/pull/125340#issuecomment-2130441817

Effectively, we can take advantage of the fact that ASCII only needs 7 bits to make the eighth bit store whether the value should be escaped or not. This adds a 256-byte lookup table, but 256 bytes *should* be small enough that very few people will mind, according to my probably not incontrovertible opinion.

The generated assembly isn't clearly better (although has fewer branches), so, I decided to benchmark on three inputs: first on a random 200KiB, then on `/bin/cat`, then on `Cargo.toml` for this repo. In all cases, the generated code ran faster on my machine. (an old i7-8700)

But, if you want to try my benchmarking code for yourself:

<details><summary>Criterion code below. Replace <code>/home/ltdk/rustsrc</code> with the appropriate directory.</summary>

```rust
#![feature(ascii_char)]
#![feature(ascii_char_variants)]
#![feature(const_option)]
#![feature(let_chains)]
use core::ascii;
use core::ops::Range;
use criterion::{criterion_group, criterion_main, Criterion};
use rand::{thread_rng, Rng};

const HEX_DIGITS: [ascii::Char; 16] = *b"0123456789abcdef".as_ascii().unwrap();

#[inline]
const fn backslash<const N: usize>(a: ascii::Char) -> ([ascii::Char; N], Range<u8>) {
    const { assert!(N >= 2) };

    let mut output = [ascii::Char::Null; N];

    output[0] = ascii::Char::ReverseSolidus;
    output[1] = a;

    (output, 0..2)
}

#[inline]
const fn hex_escape<const N: usize>(byte: u8) -> ([ascii::Char; N], Range<u8>) {
    const { assert!(N >= 4) };

    let mut output = [ascii::Char::Null; N];

    let hi = HEX_DIGITS[(byte >> 4) as usize];
    let lo = HEX_DIGITS[(byte & 0xf) as usize];

    output[0] = ascii::Char::ReverseSolidus;
    output[1] = ascii::Char::SmallX;
    output[2] = hi;
    output[3] = lo;

    (output, 0..4)
}

#[inline]
const fn verbatim<const N: usize>(a: ascii::Char) -> ([ascii::Char; N], Range<u8>) {
    const { assert!(N >= 1) };

    let mut output = [ascii::Char::Null; N];

    output[0] = a;

    (output, 0..1)
}

/// Escapes an ASCII character.
///
/// Returns a buffer and the length of the escaped representation.
const fn escape_ascii_old<const N: usize>(byte: u8) -> ([ascii::Char; N], Range<u8>) {
    const { assert!(N >= 4) };

    match byte {
        b'\t' => backslash(ascii::Char::SmallT),
        b'\r' => backslash(ascii::Char::SmallR),
        b'\n' => backslash(ascii::Char::SmallN),
        b'\\' => backslash(ascii::Char::ReverseSolidus),
        b'\'' => backslash(ascii::Char::Apostrophe),
        b'\"' => backslash(ascii::Char::QuotationMark),
        0x00..=0x1F => hex_escape(byte),
        _ => match ascii::Char::from_u8(byte) {
            Some(a) => verbatim(a),
            None => hex_escape(byte),
        },
    }
}

/// Escapes an ASCII character.
///
/// Returns a buffer and the length of the escaped representation.
const fn escape_ascii_new<const N: usize>(byte: u8) -> ([ascii::Char; N], Range<u8>) {
    /// Lookup table helps us determine how to display character.
    ///
    /// Since ASCII characters will always be 7 bits, we can exploit this to store the 8th bit to
    /// indicate whether the result is escaped or unescaped.
    ///
    /// We additionally use 0x80 (escaped NUL character) to indicate hex-escaped bytes, since
    /// escaped NUL will not occur.
    const LOOKUP: [u8; 256] = {
        let mut arr = [0; 256];
        let mut idx = 0;
        loop {
            arr[idx as usize] = match idx {
                // use 8th bit to indicate escaped
                b'\t' => 0x80 | b't',
                b'\r' => 0x80 | b'r',
                b'\n' => 0x80 | b'n',
                b'\\' => 0x80 | b'\\',
                b'\'' => 0x80 | b'\'',
                b'"' => 0x80 | b'"',

                // use NUL to indicate hex-escaped
                0x00..=0x1F | 0x7F..=0xFF => 0x80 | b'\0',

                _ => idx,
            };
            if idx == 255 {
                break;
            }
            idx += 1;
        }
        arr
    };

    let lookup = LOOKUP[byte as usize];

    // 8th bit indicates escape
    let lookup_escaped = lookup & 0x80 != 0;

    // SAFETY: We explicitly mask out the eighth bit to get a 7-bit ASCII character.
    let lookup_ascii = unsafe { ascii::Char::from_u8_unchecked(lookup & 0x7F) };

    if lookup_escaped {
        // NUL indicates hex-escaped
        if matches!(lookup_ascii, ascii::Char::Null) {
            hex_escape(byte)
        } else {
            backslash(lookup_ascii)
        }
    } else {
        verbatim(lookup_ascii)
    }
}

fn escape_bytes(bytes: &[u8], f: impl Fn(u8) -> ([ascii::Char; 4], Range<u8>)) -> Vec<ascii::Char> {
    let mut vec = Vec::new();
    for b in bytes {
        let (buf, range) = f(*b);
        vec.extend_from_slice(&buf[range.start as usize..range.end as usize]);
    }
    vec
}

pub fn criterion_benchmark(c: &mut Criterion) {
    let mut group = c.benchmark_group("escape_ascii");

    group.sample_size(1000);

    let rand_200k = &mut [0; 200 * 1024];
    thread_rng().fill(&mut rand_200k[..]);
    let cat = include_bytes!("/bin/cat");
    let cargo_toml = include_bytes!("/home/ltdk/rustsrc/Cargo.toml");

    group.bench_function("old_rand", |b| {
        b.iter(|| escape_bytes(rand_200k, escape_ascii_old));
    });
    group.bench_function("new_rand", |b| {
        b.iter(|| escape_bytes(rand_200k, escape_ascii_new));
    });

    group.bench_function("old_bin", |b| {
        b.iter(|| escape_bytes(cat, escape_ascii_old));
    });
    group.bench_function("new_bin", |b| {
        b.iter(|| escape_bytes(cat, escape_ascii_new));
    });

    group.bench_function("old_cargo_toml", |b| {
        b.iter(|| escape_bytes(cargo_toml, escape_ascii_old));
    });
    group.bench_function("new_cargo_toml", |b| {
        b.iter(|| escape_bytes(cargo_toml, escape_ascii_new));
    });

    group.finish();
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);
```

</details>

My benchmark results:

```
escape_ascii/old_rand   time:   [1.6965 ms 1.7006 ms 1.7053 ms]
Found 22 outliers among 1000 measurements (2.20%)
  4 (0.40%) high mild
  18 (1.80%) high severe
escape_ascii/new_rand   time:   [1.6749 ms 1.6953 ms 1.7158 ms]
Found 38 outliers among 1000 measurements (3.80%)
  38 (3.80%) high mild
escape_ascii/old_bin    time:   [224.59 µs 225.40 µs 226.33 µs]
Found 39 outliers among 1000 measurements (3.90%)
  17 (1.70%) high mild
  22 (2.20%) high severe
escape_ascii/new_bin    time:   [164.86 µs 165.63 µs 166.58 µs]
Found 107 outliers among 1000 measurements (10.70%)
  43 (4.30%) high mild
  64 (6.40%) high severe
escape_ascii/old_cargo_toml
                        time:   [23.397 µs 23.699 µs 24.014 µs]
Found 204 outliers among 1000 measurements (20.40%)
  21 (2.10%) high mild
  183 (18.30%) high severe
escape_ascii/new_cargo_toml
                        time:   [16.404 µs 16.438 µs 16.483 µs]
Found 88 outliers among 1000 measurements (8.80%)
  56 (5.60%) high mild
  32 (3.20%) high severe
```

Random: 1.7006ms => 1.6953ms (<1% speedup)
Binary: 225.40µs => 165.63µs (26% speedup)
Text: 23.699µs => 16.438µs (30% speedup)
2024-10-13 14:05:50 +00:00
.github Rollup merge of #131358 - onur-ozkan:129528, r=Mark-Simulacrum 2024-10-12 21:38:36 -05:00
compiler Rollup merge of #131086 - printfn:update-unicode-width, r=Mark-Simulacrum 2024-10-13 12:32:16 +02:00
library Auto merge of #125679 - clarfonthey:escape_ascii, r=joboet 2024-10-13 14:05:50 +00:00
LICENSES Rename LICENSES/GPL-2.0.txt into LICENSES/GPL-2.0-only.txt 2024-09-06 16:01:46 +02:00
src Rollup merge of #131642 - jieyouxu:build-fail-check-fail, r=Kobzol 2024-10-13 12:32:19 +02:00
tests Rollup merge of #131591 - matthiaskrgr:crashtests, r=jieyouxu 2024-10-12 21:38:37 -05:00
.clang-format Add .clang-format 2024-06-26 05:56:00 +08:00
.editorconfig Only use max_line_length = 100 for *.rs 2023-07-10 15:18:36 -07:00
.git-blame-ignore-revs Add rustfmt 2024 reformatting to git blame ignore 2024-09-23 10:02:04 +02:00
.gitattributes Rename config.toml.example to config.example.toml 2023-03-11 14:10:00 -08:00
.gitignore add nix files to gitignore 2024-10-03 11:09:44 +02:00
.gitmodules Update to LLVM 19.1.0 2024-09-20 14:41:36 -07:00
.ignore Add .ignore file to make config.toml searchable in vscode 2024-06-24 10:15:16 +02:00
.mailmap Update .mailmap 2024-10-10 12:35:26 -07:00
Cargo.lock Update unicode-width to 0.2.0 2024-10-12 21:57:50 +00:00
Cargo.toml Cleanup 2024-09-27 14:04:23 -04:00
CODE_OF_CONDUCT.md Remove the code of conduct; instead link https://www.rust-lang.org/conduct.html 2019-10-05 22:55:19 +02:00
config.example.toml bootstrap: add std_features config 2024-10-06 09:59:10 +05:30
configure Ensure ./configure works when configure.py path contains spaces 2024-02-16 18:57:22 +00:00
CONTRIBUTING.md fix: Update CONTRIBUTING.md recommend -> recommended 2023-11-16 23:57:09 +05:30
COPYRIGHT Update COPYRIGHT file 2022-10-30 10:23:14 -04:00
INSTALL.md add clarity for custom path installation 2024-10-06 07:37:00 -05:00
LICENSE-APACHE Remove appendix from LICENCE-APACHE 2019-12-30 14:25:53 +00:00
LICENSE-MIT LICENSE-MIT: Remove inaccurate (misattributed) copyright notice 2017-07-26 16:51:58 -07:00
README.md Use SVG logos in the README.md. 2024-04-03 19:48:20 +02:00
RELEASES.md Rollup merge of #129995 - alexcrichton:remove-wasm32-wasip2-release-notes, r=pietroalbini 2024-09-06 07:33:58 +02:00
REUSE.toml Fix licensing information in REUSE.toml 2024-09-06 16:01:47 +02:00
rust-bors.toml Increase timeout for new bors bot 2024-03-13 08:31:07 +01:00
rustfmt.toml Use style_edition over version 2024-09-22 19:11:28 -04:00
triagebot.toml Rollup merge of #131497 - jieyouxu:spin, r=onur-ozkan 2024-10-11 12:21:06 +02:00
x Make x capable of resolving symlinks 2023-10-14 17:53:33 +03:00
x.ps1 use & instead of start-process in x.ps1 2023-12-09 09:46:16 -05:00
x.py Fix recent python linting errors 2023-08-02 04:40:28 -04:00

README.md

This is the main source code repository for Rust. It contains the compiler, standard library, and documentation.

Why Rust?

  • Performance: Fast and memory-efficient, suitable for critical services, embedded devices, and easily integrate with other languages.

  • Reliability: Our rich type system and ownership model ensure memory and thread safety, reducing bugs at compile-time.

  • Productivity: Comprehensive documentation, a compiler committed to providing great diagnostics, and advanced tooling including package manager and build tool (Cargo), auto-formatter (rustfmt), linter (Clippy) and editor support (rust-analyzer).

Quick Start

Read "Installation" from The Book.

Installing from Source

If you really want to install from source (though this is not recommended), see INSTALL.md.

Getting Help

See https://www.rust-lang.org/community for a list of chat platforms and forums.

Contributing

See CONTRIBUTING.md.

License

Rust is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.

See LICENSE-APACHE, LICENSE-MIT, and COPYRIGHT for details.

Trademark

The Rust Foundation owns and protects the Rust and Cargo trademarks and logos (the "Rust Trademarks").

If you want to use these names or brands, please read the media guide.

Third-party logos may be subject to third-party copyrights and trademarks. See Licenses for details.