Lossless Parsing

​

What is Lossless Parsing?

1. Perfect Round-Trip: You can parse code to an AST and convert it back to get the exact original code
2. Style Preservation: Your formatting choices, indentation, and spacing are maintained
3. Comment Retention: All comments (single-line and multi-line) are kept with proper positioning
4. Token Fidelity: Every character in your source code is accounted for in the parsed representation

​

Why Lossless Parsing Matters

• Format code automatically (like rustfmt or prettier)
• Refactor code en masse (like jscodeshift)
• Provide accurate LSP features (go-to-definition, hover information)
• Perform static analysis without losing context (like Luacheck)
• Add or modify code while respecting existing style

​

How It Works

​

1. Tokenization Layer

pub struct TokenReference {
    pub(crate) leading_trivia: Vec<Token>,
    pub(crate) token: Token,
    pub(crate) trailing_trivia: Vec<Token>,
}

• Whitespace (spaces, tabs, newlines)
• Single-line comments (-- comment)
• Multi-line comments (--[[ comment ]])

​

2. AST Layer

​

Example: Preserving Style

local x = 1  -- important variable
local   y   =   2  -- lots of spaces

local x = 1
local y = 2

• The exact spacing around =
• The comment and its position
• The multiple spaces in the second line

​

Parsing with Full Moon

use full_moon::parse;

let code = "local x = 1 -- comment";
let ast = parse(code)?;

// Convert back to string - identical to original!
assert_eq!(ast.to_string(), code);

/// Creates an [`Ast`](ast::Ast) from Lua code.
/// Will use the most complete set of Lua versions enabled in your feature set.
///
/// # Errors
/// If the code passed cannot be tokenized, a TokenizerError will be returned.
/// If the code passed is not valid Lua 5.1 code, an AstError will be returned,
/// specifically AstError::UnexpectedToken.
///
/// ```rust
/// assert!(full_moon::parse("local x = 1").is_ok());
/// assert!(full_moon::parse("local x = ").is_err());
/// ```
pub fn parse(code: &str) -> Result<ast::Ast, Vec<Error>> {
    parse_fallible(code, LuaVersion::new()).into_result()
}

​

Trivia Types

TokenType::Whitespace {
    characters: ShortString,
}

​

Position Tracking

pub struct Position {
    pub(crate) bytes: usize,
    pub(crate) line: usize,
    pub(crate) character: usize,
}

impl Position {
    /// How many bytes, ignoring lines, it would take to find this position
    pub fn bytes(self) -> usize
    
    /// Index of the character on the line for this position
    pub fn character(self) -> usize
    
    /// Line the position lies on
    pub fn line(self) -> usize
}

• Report error locations accurately
• Implement “go to definition” features
• Create precise source maps
• Track changes across edits

​

Use Cases

1. Code Formatters: Reformat code while preserving intentional style choices
2. Refactoring Tools: Rename variables or restructure code without breaking formatting
3. Static Analysis: Analyze code and report issues with exact locations
4. Code Generation: Insert generated code that matches surrounding style
5. Documentation Tools: Extract comments and associate them with code elements

​

Comparison: Lossy vs Lossless

​

Next Steps