compiler optimization output
This commit is contained in:
@@ -56,6 +56,7 @@ Modules loaded with `use()`:
|
||||
## Tools
|
||||
|
||||
- [**Command Line**](/docs/cli/) — the `pit` tool
|
||||
- [**Testing**](/docs/testing/) — writing and running tests
|
||||
- [**Writing C Modules**](/docs/c-modules/) — native extensions
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -78,12 +78,13 @@ pit build
|
||||
|
||||
### pit test
|
||||
|
||||
Run tests.
|
||||
Run tests. See [Testing](/docs/testing/) for the full guide.
|
||||
|
||||
```bash
|
||||
pit test # run tests in current package
|
||||
pit test all # run all tests
|
||||
pit test <package> # run tests in specific package
|
||||
pit test suite --verify --diff # with IR verification and differential testing
|
||||
```
|
||||
|
||||
### pit link
|
||||
|
||||
264
docs/compiler-tools.md
Normal file
264
docs/compiler-tools.md
Normal file
@@ -0,0 +1,264 @@
|
||||
---
|
||||
title: "Compiler Inspection Tools"
|
||||
description: "Tools for inspecting and debugging the compiler pipeline"
|
||||
weight: 50
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
ƿit includes a set of tools for inspecting the compiler pipeline at every stage. These are useful for debugging, testing optimizations, and understanding what the compiler does with your code.
|
||||
|
||||
## Pipeline Overview
|
||||
|
||||
The compiler runs in stages:
|
||||
|
||||
```
|
||||
source → tokenize → parse → fold → mcode → streamline → output
|
||||
```
|
||||
|
||||
Each stage has a corresponding dump tool that lets you see its output.
|
||||
|
||||
| Stage | Tool | What it shows |
|
||||
|-------------|-------------------|----------------------------------------|
|
||||
| fold | `dump_ast.cm` | Folded AST as JSON |
|
||||
| mcode | `dump_mcode.cm` | Raw mcode IR before optimization |
|
||||
| streamline | `dump_stream.cm` | Before/after instruction counts + IR |
|
||||
| streamline | `dump_types.cm` | Optimized IR with type annotations |
|
||||
| streamline | `streamline.ce` | Full optimized IR as JSON |
|
||||
| all | `ir_report.ce` | Structured optimizer flight recorder |
|
||||
|
||||
All tools take a source file as input and run the pipeline up to the relevant stage.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# see raw mcode IR
|
||||
./cell --core . dump_mcode.cm myfile.ce
|
||||
|
||||
# see what the optimizer changed
|
||||
./cell --core . dump_stream.cm myfile.ce
|
||||
|
||||
# full optimizer report with events
|
||||
./cell --core . ir_report.ce --full myfile.ce
|
||||
```
|
||||
|
||||
## dump_ast.cm
|
||||
|
||||
Prints the folded AST as JSON. This is the output of the parser and constant folder, before mcode generation.
|
||||
|
||||
```bash
|
||||
./cell --core . dump_ast.cm <file.ce|file.cm>
|
||||
```
|
||||
|
||||
## dump_mcode.cm
|
||||
|
||||
Prints the raw mcode IR before any optimization. Shows the instruction array as formatted text with opcode, operands, and program counter.
|
||||
|
||||
```bash
|
||||
./cell --core . dump_mcode.cm <file.ce|file.cm>
|
||||
```
|
||||
|
||||
## dump_stream.cm
|
||||
|
||||
Shows a before/after comparison of the optimizer. For each function, prints:
|
||||
- Instruction count before and after
|
||||
- Number of eliminated instructions
|
||||
- The streamlined IR (nops hidden by default)
|
||||
|
||||
```bash
|
||||
./cell --core . dump_stream.cm <file.ce|file.cm>
|
||||
```
|
||||
|
||||
## dump_types.cm
|
||||
|
||||
Shows the optimized IR with type annotations. Each instruction is followed by the known types of its slot operands, inferred by walking the instruction stream.
|
||||
|
||||
```bash
|
||||
./cell --core . dump_types.cm <file.ce|file.cm>
|
||||
```
|
||||
|
||||
## streamline.ce
|
||||
|
||||
Runs the full pipeline (tokenize, parse, fold, mcode, streamline) and outputs the optimized IR as JSON. Useful for piping to `jq` or saving for comparison.
|
||||
|
||||
```bash
|
||||
./cell --core . streamline.ce <file.ce|file.cm>
|
||||
```
|
||||
|
||||
## ir_report.ce
|
||||
|
||||
The optimizer flight recorder. Runs the full pipeline with structured logging and outputs machine-readable, diff-friendly JSON. This is the most detailed tool for understanding what the optimizer did and why.
|
||||
|
||||
```bash
|
||||
./cell --core . ir_report.ce [options] <file.ce|file.cm>
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--summary` | Per-pass JSON summaries with instruction counts and timing (default) |
|
||||
| `--events` | Include rewrite events showing each optimization applied |
|
||||
| `--types` | Include type delta records showing inferred slot types |
|
||||
| `--ir-before=PASS` | Print canonical IR before a specific pass |
|
||||
| `--ir-after=PASS` | Print canonical IR after a specific pass |
|
||||
| `--ir-all` | Print canonical IR before and after all passes |
|
||||
| `--full` | Everything: summary + events + types + ir-all |
|
||||
|
||||
With no flags, `--summary` is the default.
|
||||
|
||||
### Output Format
|
||||
|
||||
Output is line-delimited JSON. Each line is a self-contained JSON object with a `type` field:
|
||||
|
||||
**`type: "pass"`** — Per-pass summary with categorized instruction counts before and after:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "pass",
|
||||
"pass": "eliminate_type_checks",
|
||||
"fn": "fib",
|
||||
"ms": 0.12,
|
||||
"changed": true,
|
||||
"before": {"instr": 77, "nop": 0, "guard": 16, "branch": 28, ...},
|
||||
"after": {"instr": 77, "nop": 1, "guard": 15, "branch": 28, ...},
|
||||
"changes": {"guards_removed": 1, "nops_added": 1}
|
||||
}
|
||||
```
|
||||
|
||||
**`type: "event"`** — Individual rewrite event with before/after instructions and reasoning:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "event",
|
||||
"pass": "eliminate_type_checks",
|
||||
"rule": "incompatible_type_forces_jump",
|
||||
"at": 3,
|
||||
"before": [["is_int", 5, 2, 4, 9], ["jump_false", 5, "rel_ni_2", 4, 9]],
|
||||
"after": ["_nop_tc_1", ["jump", "rel_ni_2", 4, 9]],
|
||||
"why": {"slot": 2, "known_type": "float", "checked_type": "int"}
|
||||
}
|
||||
```
|
||||
|
||||
**`type: "types"`** — Inferred type information for a function:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "types",
|
||||
"fn": "fib",
|
||||
"param_types": {},
|
||||
"slot_types": {"25": "null"}
|
||||
}
|
||||
```
|
||||
|
||||
**`type: "ir"`** — Canonical IR text for a function at a specific point:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "ir",
|
||||
"when": "before",
|
||||
"pass": "all",
|
||||
"fn": "fib",
|
||||
"text": "fn fib (args=1, slots=26)\n @0 access s2, 2\n ..."
|
||||
}
|
||||
```
|
||||
|
||||
### Rewrite Rules
|
||||
|
||||
Each pass records events with named rules:
|
||||
|
||||
**eliminate_type_checks:**
|
||||
- `known_type_eliminates_guard` — type already known, guard removed
|
||||
- `incompatible_type_forces_jump` — type conflicts, conditional jump becomes unconditional
|
||||
- `num_subsumes_int_float` — num check satisfied by int or float
|
||||
- `dynamic_to_field` — load_dynamic/store_dynamic narrowed to field access
|
||||
- `dynamic_to_index` — load_dynamic/store_dynamic narrowed to index access
|
||||
|
||||
**simplify_algebra:**
|
||||
- `add_zero`, `sub_zero`, `mul_one`, `div_one` — identity operations become moves
|
||||
- `mul_zero` — multiplication by zero becomes constant
|
||||
- `self_eq`, `self_ne` — same-slot comparisons become constants
|
||||
|
||||
**simplify_booleans:**
|
||||
- `not_jump_false_fusion` — not + jump_false fused into jump_true
|
||||
- `not_jump_true_fusion` — not + jump_true fused into jump_false
|
||||
- `double_not` — not + not collapsed to move
|
||||
|
||||
**eliminate_moves:**
|
||||
- `self_move` — move to same slot becomes nop
|
||||
|
||||
**eliminate_dead_jumps:**
|
||||
- `jump_to_next` — jump to immediately following label becomes nop
|
||||
|
||||
### Canonical IR Format
|
||||
|
||||
The `--ir-all`, `--ir-before`, and `--ir-after` flags produce a deterministic text representation of the IR:
|
||||
|
||||
```
|
||||
fn fib (args=1, slots=26)
|
||||
@0 access s2, 2
|
||||
@1 is_int s4, s1 ; [guard]
|
||||
@2 jump_false s4, "rel_ni_2" ; [branch]
|
||||
@3 --- nop (tc) ---
|
||||
@4 jump "rel_ni_2" ; [branch]
|
||||
@5 lt_int s3, s1, s2
|
||||
@6 jump "rel_done_4" ; [branch]
|
||||
rel_ni_2:
|
||||
@8 is_num s4, s1 ; [guard]
|
||||
```
|
||||
|
||||
Properties:
|
||||
- `@N` is the raw array index, stable across passes (passes replace, never insert or delete)
|
||||
- `sN` prefix distinguishes slot operands from literal values
|
||||
- String operands are quoted
|
||||
- Labels appear as indented headers with a colon
|
||||
- Category tags in brackets: `[guard]`, `[branch]`, `[load]`, `[store]`, `[call]`, `[arith]`, `[move]`, `[const]`
|
||||
- Nops shown as `--- nop (reason) ---` with reason codes: `tc` (type check), `bl` (boolean), `mv` (move), `dj` (dead jump), `ur` (unreachable)
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# what passes changed something?
|
||||
./cell --core . ir_report.ce --summary myfile.ce | jq 'select(.changed)'
|
||||
|
||||
# list all rewrite rules that fired
|
||||
./cell --core . ir_report.ce --events myfile.ce | jq 'select(.type == "event") | .rule'
|
||||
|
||||
# diff IR before and after optimization
|
||||
./cell --core . ir_report.ce --ir-all myfile.ce | jq -r 'select(.type == "ir") | .text'
|
||||
|
||||
# full report for analysis
|
||||
./cell --core . ir_report.ce --full myfile.ce > report.json
|
||||
```
|
||||
|
||||
## ir_stats.cm
|
||||
|
||||
A utility module used by `ir_report.ce` and available for custom tooling. Not a standalone tool.
|
||||
|
||||
```javascript
|
||||
var ir_stats = use("ir_stats")
|
||||
|
||||
ir_stats.detailed_stats(func) // categorized instruction counts
|
||||
ir_stats.ir_fingerprint(func) // djb2 hash of instruction array
|
||||
ir_stats.canonical_ir(func, name, opts) // deterministic text representation
|
||||
ir_stats.type_snapshot(slot_types) // frozen copy of type map
|
||||
ir_stats.type_delta(before_types, after_types) // compute type changes
|
||||
ir_stats.category_tag(op) // classify an opcode
|
||||
```
|
||||
|
||||
### Instruction Categories
|
||||
|
||||
`detailed_stats` classifies each instruction into one of these categories:
|
||||
|
||||
| Category | Opcodes |
|
||||
|----------|---------|
|
||||
| load | `load_field`, `load_index`, `load_dynamic`, `get`, `access` (non-constant) |
|
||||
| store | `store_field`, `store_index`, `store_dynamic`, `set_var`, `put`, `push` |
|
||||
| branch | `jump`, `jump_true`, `jump_false`, `jump_not_null` |
|
||||
| call | `invoke`, `goinvoke` |
|
||||
| guard | `is_int`, `is_text`, `is_num`, `is_bool`, `is_null`, `is_array`, `is_func`, `is_record`, `is_stone` |
|
||||
| arith | `add_int`, `sub_int`, ..., `add_float`, ..., `concat`, `neg_int`, `neg_float`, bitwise ops |
|
||||
| move | `move` |
|
||||
| const | `int`, `true`, `false`, `null`, `access` (with constant value) |
|
||||
| label | string entries that are not nops |
|
||||
| nop | strings starting with `_nop_` |
|
||||
| other | everything else (`frame`, `setarg`, `array`, `record`, `function`, `return`, etc.) |
|
||||
170
docs/testing.md
Normal file
170
docs/testing.md
Normal file
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: "Testing"
|
||||
description: "Writing and running tests in ƿit"
|
||||
weight: 45
|
||||
type: "docs"
|
||||
---
|
||||
|
||||
ƿit has built-in support for writing and running tests. Tests live in the `tests/` directory of a package and are `.cm` modules that return a record of test functions.
|
||||
|
||||
## Writing Tests
|
||||
|
||||
A test file returns a record where each key starting with `test_` is a test function. A test passes if it returns `null` (or nothing). It fails if it returns a text string describing the failure.
|
||||
|
||||
```javascript
|
||||
// tests/math.cm
|
||||
return {
|
||||
test_addition: function() {
|
||||
if (1 + 2 != 3) return "expected 3"
|
||||
},
|
||||
|
||||
test_division: function() {
|
||||
if (10 / 3 != 3.333333333333333333) return "unexpected result"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Test functions take no arguments. Use early returns with a failure message to report errors:
|
||||
|
||||
```javascript
|
||||
test_array_push: function() {
|
||||
var a = [1, 2]
|
||||
a[] = 3
|
||||
if (length(a) != 3) return "expected length 3, got " + text(length(a))
|
||||
if (a[2] != 3) return "expected a[2] to be 3"
|
||||
}
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
pit test # run all tests in current package
|
||||
pit test suite # run a specific test file (tests/suite.cm)
|
||||
pit test tests/math # same, with explicit path
|
||||
pit test all # run all tests in current package
|
||||
pit test package <name> # run all tests in a named package
|
||||
pit test package <name> <test> # run a specific test in a named package
|
||||
pit test package all # run tests from all installed packages
|
||||
```
|
||||
|
||||
### Flags
|
||||
|
||||
```bash
|
||||
pit test suite -g # run GC after each test (useful for detecting leaks)
|
||||
pit test suite --verify # enable IR verification during compilation
|
||||
pit test suite --diff # run each test optimized and unoptimized, compare results
|
||||
```
|
||||
|
||||
`--verify` and `--diff` can be combined:
|
||||
|
||||
```bash
|
||||
pit test suite --verify --diff
|
||||
```
|
||||
|
||||
## IR Verification
|
||||
|
||||
The `--verify` flag enables structural validation of the compiler's intermediate representation after each optimizer pass. This catches bugs like invalid slot references, broken jump targets, and malformed instructions.
|
||||
|
||||
When verification fails, errors are printed with the pass name that introduced them:
|
||||
|
||||
```
|
||||
[verify_ir] slot_bounds: slot 12 out of range 0..9 in instruction add_int
|
||||
[verify_ir] 1 errors after dead_code_elimination
|
||||
```
|
||||
|
||||
IR verification adds overhead and is intended for development, not production use.
|
||||
|
||||
## Differential Testing
|
||||
|
||||
Differential testing runs each test through two paths — with the optimizer enabled and with it disabled — and compares results. Any mismatch between the two indicates an optimizer bug.
|
||||
|
||||
### Inline Mode
|
||||
|
||||
The `--diff` flag on `pit test` runs each test module through both paths during a normal test run:
|
||||
|
||||
```bash
|
||||
pit test suite --diff
|
||||
```
|
||||
|
||||
Output includes a mismatch count at the end:
|
||||
|
||||
```
|
||||
Tests: 493 passed, 0 failed, 493 total
|
||||
Diff mismatches: 0
|
||||
```
|
||||
|
||||
### Standalone Mode
|
||||
|
||||
`pit diff` is a dedicated differential testing tool with detailed mismatch reporting:
|
||||
|
||||
```bash
|
||||
pit diff # diff all test files in current package
|
||||
pit diff suite # diff a specific test file
|
||||
pit diff tests/math # same, with explicit path
|
||||
```
|
||||
|
||||
For each test function, it reports whether the optimized and unoptimized results match:
|
||||
|
||||
```
|
||||
tests/suite.cm: 493 passed, 0 failed
|
||||
----------------------------------------
|
||||
Diff: 493 passed, 0 failed, 493 total
|
||||
```
|
||||
|
||||
When a mismatch is found:
|
||||
|
||||
```
|
||||
tests/suite.cm: 492 passed, 1 failed
|
||||
MISMATCH: test_foo: result mismatch opt=42 noopt=43
|
||||
```
|
||||
|
||||
## Fuzz Testing
|
||||
|
||||
The fuzzer generates random self-checking programs, compiles them, and runs them through both optimized and unoptimized paths. Each generated program contains test functions that validate their own expected results, so failures catch both correctness bugs and optimizer mismatches.
|
||||
|
||||
```bash
|
||||
pit fuzz # 100 iterations, random seed
|
||||
pit fuzz 500 # 500 iterations, random seed
|
||||
pit fuzz --seed 42 # 100 iterations, deterministic seed
|
||||
pit fuzz 1000 --seed 42 # 1000 iterations, deterministic seed
|
||||
```
|
||||
|
||||
The fuzzer generates programs that exercise:
|
||||
|
||||
- Integer and float arithmetic with known expected results
|
||||
- Control flow (if/else, while loops)
|
||||
- Closures and captured variable mutation
|
||||
- Records and property access
|
||||
- Arrays and iteration
|
||||
- Higher-order functions
|
||||
- Disruption handling
|
||||
- Text concatenation
|
||||
|
||||
On failure, the generated source is saved to `tests/fuzz_failures/` for reproduction:
|
||||
|
||||
```
|
||||
Fuzzing: 1000 iterations, starting seed=42
|
||||
FAIL seed=57: diff fuzz_3: opt=10 noopt=11
|
||||
saved to tests/fuzz_failures/seed_57.cm
|
||||
----------------------------------------
|
||||
Fuzz: 999 passed, 1 failed, 1000 total
|
||||
Failures saved to tests/fuzz_failures/
|
||||
```
|
||||
|
||||
Saved failure files are valid `.cm` modules that can be run directly or added to the test suite.
|
||||
|
||||
## Test File Organization
|
||||
|
||||
Tests live in the `tests/` directory of a package:
|
||||
|
||||
```
|
||||
mypackage/
|
||||
├── pit.toml
|
||||
├── math.cm
|
||||
└── tests/
|
||||
├── suite.cm # main test suite
|
||||
├── math.cm # math-specific tests
|
||||
└── disrupt.cm # disruption tests
|
||||
```
|
||||
|
||||
All `.cm` files under `tests/` are discovered automatically by `pit test`.
|
||||
@@ -1,3 +1,7 @@
|
||||
// dump_ast.cm — pretty-print the folded AST as JSON
|
||||
//
|
||||
// Usage: ./cell --core . dump_ast.cm <file.ce|file.cm>
|
||||
|
||||
var fd = use("fd")
|
||||
var json = use("json")
|
||||
var tokenize = use("tokenize")
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
// streamline.ce — run the full compile + optimize pipeline, output JSON
|
||||
//
|
||||
// Usage: ./cell --core . streamline.ce <file.ce|file.cm>
|
||||
|
||||
var fd = use("fd")
|
||||
var json = use("json")
|
||||
var tokenize = use("tokenize")
|
||||
|
||||
Reference in New Issue
Block a user