Compiler - Core & Environment

The bytecode compiler is responsible for translating the AST (Abstract Syntax Tree) into a flattened array of OpCode instructions wrapped in a Chunk. It runs after the parser, expander, and semantic checker. The core compiler structures and environment abstractions are defined in src/compiler/.

Module Layout

Compiler Struct

The top-level Compiler orchestrates the entire translation pipeline. It owns the global environment (functions, variables, constants) that spans the entire program.

• globals: Maps global variable names (StringId) to an integer index 0..N.
• func_indices: Maps function/fiber names to their index in the functions array.
• functions: Accumulates JIT-ready Chunks representing compiled functions/fibers. Each Chunk carries param_types: Vec<TypeTag> filled from the declared parameter signatures; the JIT seeds parameter registers from it during type inference.
• constants: A deduplicated pool of compile-time constants (strings, floats, ints, maps) used by OpCode::LoadConst.
• disable_inline: A single flag set once before compiling the whole program (not per CompileContext), which - when true - skips the HIR inlining pass entirely. See "Compilation Dispatch" below and compiler/hir/hir_inline.md.

Pre-Registration (globals.rs)

Before any user bytecodes are emitted (the built-in module preload described below is emitted first), Compiler::compile calls globals::register_globals_recursive. This performs a pre-pass over the AST to assign indices to:

1. Functions and fibers (func_indices, pushing skeleton Chunks into functions).

2. Global variables (globals).

3. Database instances.

Built-in modules (json, date, random, store, input) are hardcoded as global indices 0..4 (inclusive, totally 5 elements). The compiler synthesizes bytecode in the main chunk to preload these globals via LoadConst and SetVar.

CompileContext

A short-lived mutable view passed to compilation routines. It decouples the global state from specific function compilations, allowing recursive or independent function compilation without borrowing issues.

FunctionCompiler Struct

Compiles a single function, fiber, or the top-level main program. Shared identically by both the AST-to-bytecode path and the HIR-to-bytecode path (hir::compile_hir_to_chunk) - see compiler/hir/hir_codegen.md.

• bytecode / spans: The parallel arrays of instructions and debug spans that will form the final Chunk.
• scopes: A stack of lexical scopes binding variable names maps to local register indices. Pushed on {, popped on }.
• next_local: The next free local register index in the current frame.
• max_locals_used: The high-water mark of next_local, dictating the size of the runtime stack frame overhead. Kept in sync via sync_max_locals(), called after push_reg() and after any operation that advances next_local directly.
• loop_stack: Tracks control flow for break and continue as a stack of typed LoopFrame values (start_pc, breaks, continues, fiber_reg) - see compiler/compiler/compiler_stmt.md for the full structure and its fields.
• local_regs: The set of register indices permanently reserved for named locals (as opposed to transient registers allocated by push_reg/pop_reg during subexpression compilation). Populated up front when compiling from HIR, where every local already has a fixed index.
• inline_stack / inline_result_locals: Used when compiling an HIR InlineBlock (the result of function inlining, see compiler/hir/hir_inline.md) - they track, for each nested inlined call currently being compiled, which local register should receive the inlined function's return value.

Register Allocation

The compiler treats arguments and local variables equally as u8 local registers.

• push_reg(): Allocates the next sequential register and bumps max_locals_used.
• pop_reg(): Reclaims the most recently pushed register.

Register reuse is resolved statically using lexical scopes and scope-based lifetime analysis during the lowering phase to minimize local variable index allocation. (Note: The speculative RegisterManager optimization pass was deleted in Phase 3A as scope-based reuse was sufficient).

Compilation Dispatch

Compiler::compile iterates over the AST, compiling main statements into the main FunctionCompiler, and delegating FunctionDef and FiberDef to independent compilation calls which produce one Chunk per function.

For top-level functions and fibers, compilation goes through an intermediate HIR stage rather than straight from AST to bytecode: after pre-registration, Compiler::compile calls hir::lower_program to lower every FunctionDef/FiberDef body to HIR, then (unless self.disable_inline is set) hir::run_inliner_pass to inline eligible call sites within that HIR. Each function is then compiled to its Chunk via hir::compile_hir_to_chunk. If a function has no corresponding entry in the lowered HIR map (which should not normally happen, but is handled defensively), compilation falls back to compile_function_helper, the direct AST-to-bytecode path described in compiler/compiler/compiler_stmt.md and compiler/compiler/compiler_expr.md. Both paths produce the same OpCode output format and share the same FunctionCompiler/CompileContext machinery - see compiler/hir/README.md for the HIR pipeline itself.

The resulting output is a tuple: (MainChunk, ConstantsPool, FunctionsArray).

Default Value Generation & Constant Pool Edge Cases

When variables are declared without an explicit value (e.g., i: count;), the compiler automatically calls defaults::get_default_value and pushes a LoadConst instruction.

Limitation / Nuance: For reference types like Arrays, Sets, Maps, and Tables, the default value initialized in the constant pool is a single shared Arc<RwLock<...>> instance. The executor runtime must therefore ensure to deeply clone or uniquely instantiate these defaults upon variable assignment, otherwise all uninitialized collections would point to the same memory block.

Table Literal Skeletons (compile_table.rs)

Defining table { ... } literals triggers a highly unique compiler path:

1. The compiler analyzes the schema AST (columns).

2. It statically builds an empty TableObj containing the exact VMColumn types, primary keys, and auto-increments.

3. It inserts this empty, fully-formed table into the constant pool.

4. It emits TableInit holding the index to this "skeleton". The runtime simply clones the skeleton rather than parsing schema metadata, heavily boosting JIT speed.

Upvalue Capture & Closures (upvalue.rs / scope_tracker.rs)

Constraint: The XCX compiler captures variables flatly.

scope_tracker.rs uses a mechanism where if a local variable is not found in the current frame, it checks parent_locals. If it exists there, it is pushed into a captures vector and implicitly assigned a local slot mapped as 1 + captures_index.

This results in a tightly bounded one-level closure structure; deeply nested closures require manual parameter propagation or run the risk of losing scope tracking.

Backpatching

When compiling jumps (e.g., if, while, or, and), the jump target instruction pointer (IP) is not known until the body has been compiled. The compiler emits placeholder instructions, records their IPs, and backpatches them inline during bytecode emission (e.g. inside compile_control.rs). (Note: The separate patch.rs helper has been retired and backpatching logic is resolved inline).