DOCUMENTATION

Lexer

The Lexer transforms the raw source text of XCX into a flat stream of Token values. It is a single-pass, byte-oriented scanner defined in src/frontend/lexer/.

Module Layout

Cursor

The cursor operates on raw bytes for speed. It handles UTF-8 only to track char_pos and recognize multi-byte set operation symbols (∪, ∩, ⊕). All ASCII tokens are processed byte-by-byte.

Key methods:

Token

Span carries line, col, and len (in characters, not bytes). Created via Token::new(kind, line, col, len).

TokenKind

All variants of the TokenKind enum:

Keywords - Types

Keywords - Control Flow

Keywords - Logic

Keywords - Builtins / Modules

Set Operations

Identifier and Literal Tokens

Attribute Tokens (@ prefix)

@ is not a generic precursor for arbitrary user configurations. The analyzer maps only known compiler attribute definitions. Using an unrecognized attribute returns Unknown('@').

Operators

Punctuation

Special System Actions

Lexer

Lexer::new(source: &str) loads the full target source code into its buffer. Token extraction is requested cyclically, driven by calls to next_token(&mut Interner) -> Token. The Lexer does not eagerly tokenize the entire file into an iterator; instead, it performs lazy, on-demand scanning per trigger. For memory efficiency, the Parser controls the Lexer, requesting single tokens as needed via next_token.

Token Dispatch Cycle Logic (next_token)

1. trivia::skip - Consumes all whitespace characters and skips past comments.

2. Captures the current line and col to construct a Span for error reporting and symbol tracking.

3. Checks for the end of the file, returning the EOF token if reached.

4. Examines the current byte c:

• Decimal Digits - Delegates to number::lex for numeric parsing, utilizing small token backtracking prior to transferring logic flow.
• Alphabetical ASCII or _ - Delegates to Lexer::identifier. This loops over alphanumeric characters, collecting the string and dispatching it to keyword::lookup to check against known keywords vs custom variables.
• Bytes >= 128 (Non-ASCII) - Transitions to logic interpreting UTF-8 math set operators (∪, ∩, ⊕), probing prefixes against static mapping arrays. Will return Unknown if invalid.
• Double Quote " - Consumes the delimiter and passes structural handling to string_lit::lex.
• Hashtag # - Controller searches bounds for alphanumeric text block loops matching suffix tag traits, throwing the grouped logic as a Tag using the internally captured Interner index string ID.
• Operator < - Inspects the bounds for compound operators. If it catches <<<, it delegates extraction to string_lit::lex_raw, which slurps all text raw, bypassing typical escape decoding logic inside standard strings.
• Modifying Attribute @ - Bumps the parser search buffer to a small boundary of jumps matching standard compiler procedural attributes (e.g. @step, @auto). Follows strict bounded rules mapping optional behavior checks.
• Unrecognized / Isolated Operators - Falls out of logic into an exhaustive static symbol map match block to yield single or double-character relational or mathematical tokens.

Identifier and Keyword Recognition

Reads byte segments in a loop while is_ascii_alphanumeric() || c == '_' || c >= 128. The gathered structure is passed along as text, matched via a dictionary lookup assistant on the underlying keyword pool via keyword::lookup(text, interner).

The keyword::lookup system operates over two phase mappings:

1. Case-sensitive checks: Mathematical set operations (UNION, INTERSECTION), control logical operators (AND, OR, NOT, HAS), and explicit Set modifiers (N, Q, Z, S, B, C).

2. Case-insensitive checks: All other system modules and loop bindings default back to to_lowercase(), matching seamlessly against mapping checks.

Compilation flow fallback binds anything outside of mapped definitions as a programmatic user variable instance, strictly returning standard Identifier(StringId) tokens.

Numeric Lexical Analysis (number::lex)

A buffering procedure processes sequence bytes character by character until it determines the token boundaries. It executes logic checks for decimal separators (.) and delegates to optional fallback routines for method accessors. If the period is detected to be part of an object method accessor (foo.bar), it splits the token instead of treating it as a decimal point. Fully verified numeric sequences without a fractional part (1, 33) map dynamically into i64 integers, whereas values containing a fractional component are parsed under f64 boundaries.

String Literal Lexing (string_lit::lex)

This mechanism captures text sequences bounded completely by double quotation marks ". It includes a compiler assistant sequence for validating and escaping characters:

Any unrecognized escape sequence formatted as \X will safely fall back and output directly into the parsed structure as text.

Raw Block Lexing (string_lit::lex_raw)

When the scanning controller hits the <<< delimiter, it triggers an unrestricted text ingestion sequence. Precursor logic analysis evaluates and reads all raw text sequences without any string evaluation or escape token replacements, continuously caching strings until it encounters the closure operator >>>. Bounded operations are stored into system memory safely utilizing RawBlock(StringId) pointers. This avoids conventional formatting issues present inside string parsing loops (i.e. ignoring string interpolation completely and returning pure text).

Whitespace and Comments (Trivia: trivia::skip)

This skips over empty ASCII whitespace without touching the central compiler parsing rules. Furthermore, it manages commentary structures bypassing syntactical evaluations:

C-style block allocations (//, / /) intentionally trigger a hard panic fallback via the active parser verification steps to explicitly instruct the programmer about syntax migration formats into native XCX standards.

String Interning Mechanism

The lexical scanner accepts the input array and utilizes an internal &mut Interner context object, injected repeatedly over next_token procedures. Tokens that hold pure text sequences natively (e.g. Identifier, StringLiteral, RawBlock, Tag) push string tracking away from heavy memory manipulation patterns toward StringId values rather than dynamically allocated String representations. A StringId acts as a lightweight u32 array index directly tied to the overarching tracking compiler namespace accessed purely efficiently via Interner::lookup(id).

This procedural cache shares functionality seamlessly with the parser and semantic expanders, enabling fast syntax validation queries using pure integer comparisons, creating O(1) tracking arrays over the lifetime of the pipeline.

Lexer Error Handling

The lexical parser consciously avoids complex Result nesting during its extraction pass. Unknown characters and failed verifications drop instantly out of standard loops, returning a flag under Unknown(char), where standard modifier fallbacks execute pipeline triggers through the main Reporter class structure to yield explicit syntax error communications natively. Specifically, panics generated due to the misuse of C-style comment syntax include distinct error handling explanations embedded to streamline developer adoption.