918547dbe9
2911: Implement collecting errors while tokenizing r=matklad a=Veetaha
Now we are collecting errors from `rustc_lexer` and returning them in `ParsedToken { token, error }` and `ParsedTokens { tokens, errors }` structures **([UPD]: this is now simplified, see updates bellow)**.
The main changes are introduced in `ra_syntax/parsing/lexer.rs`. It now exposes the following functions and types:
```rust
pub fn tokenize(text: &str) -> ParsedTokens;
pub fn tokenize_append(text: &str, parsed_tokens_to_append_to: &mut ParsedTokens);
pub fn first_token(text: &str) -> Option<ParsedToken>; // allows any number of tokens in text
pub fn single_token(text: &str) -> Option<ParsedToken>; // allows only a single token in text
pub struct ParsedToken { pub token: Token, pub error: Option<SyntaxError> }
pub struct ParsedTokens { pub tokens: Vec<Token>, pub errors: Vec<SyntaxError> }
pub enum TokenizeError { /* Simple enum which reflects rustc_lexer tokenization errors */ }
```
In the first commit I implemented it with iterators, but then decided that since this crate is ad hoc for `rust-analyzer` and we clearly see the places of its usage it would be better to simplify it to vectors.
This is currently WIP, because I want to add tests for error messages generated by the lexer.
I'd like to listen to you thoughts how to define these tests in `ra_syntax/test-data` dir.
Related issues: #223
**[UPD]**
After the PR review the API was simplified:
```rust
pub fn tokenize(text: &str) -> (Vec<Token>, Vec<SyntaxError>);
// Both lex functions do not check for unescape errors
pub fn lex_single_syntax_kind(text: &str) -> Option<(SyntaxKind, Option<SyntaxError>)>;
pub fn lex_single_valid_syntax_kind(text: &str) -> Option<SyntaxKind>;
// This will be removed in the next PR in favour of simlifying `SyntaxError` to `(String, TextRange)`
pub enum TokenizeError { /* Simple enum which reflects rustc_lexer tokenization errors */ }
// this is private, but may be made public if such demand would exist in future (least privilege principle)
fn lex_first_token(text: &str) -> Option<(Token, Option<SyntaxError>)>;
```
Co-authored-by: Veetaha <gerzoh1@gmail.com>
|
||
|---|---|---|
| .cargo | ||
| .github | ||
| .vscode | ||
| crates | ||
| docs | ||
| editors | ||
| xtask | ||
| .gitattributes | ||
| .gitignore | ||
| bors.toml | ||
| Cargo.lock | ||
| Cargo.toml | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| README.md | ||
| rustfmt.toml | ||
Rust Analyzer is an experimental modular compiler frontend for the Rust language. It is a part of a larger rls-2.0 effort to create excellent IDE support for Rust. If you want to get involved, check the rls-2.0 working group:
https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0
Work on the Rust Analyzer is sponsored by
Language Server Quick Start
Rust Analyzer is a work-in-progress, so you'll have to build it from source, and you might encounter critical bugs. That said, it is complete enough to provide a useful IDE experience and some people use it as a daily driver.
To build rust-analyzer, you need:
- latest stable rust for language server itself
- latest stable npm and VS Code for VS Code extension
To quickly install rust-analyzer with VS Code extension with standard setup
(code and cargo in $PATH, etc), use this:
# clone the repo
$ git clone https://github.com/rust-analyzer/rust-analyzer && cd rust-analyzer
# install both the language server and VS Code extension
$ cargo xtask install
# alternatively, install only the server. Binary name is `ra_lsp_server`.
$ cargo xtask install --server
For non-standard setup of VS Code and other editors, or if the language server cannot start, see ./docs/user.
Documentation
If you want to contribute to rust-analyzer or just curious about how things work under the hood, check the ./docs/dev folder.
If you want to use rust-analyzer's language server with your editor of choice, check ./docs/user folder. It also contains some tips & tricks to help you be more productive when using rust-analyzer.
Getting in touch
We are on the rust-lang Zulip!
https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Frls-2.2E0
Quick Links
- API docs: https://rust-analyzer.github.io/rust-analyzer/ra_ide/
- Website: https://rust-analyzer.github.io/
License
Rust analyzer is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0).
See LICENSE-APACHE and LICENSE-MIT for details.