cell/fix_pipeline.md

Fix Compilation Pipeline Bootstrap

Problem

After merging fix_gc into pitweb, the compilation pipeline .cm source files (tokenize.cm, parse.cm, fold.cm, mcode.cm, streamline.cm) cannot bootstrap themselves.

The old pitweb pipeline mcode compiles the merged .cm source without errors, but the resulting new pipeline mcode is semantically broken — it can't even compile var x = 42; print(x).

Both branches worked independently. The merge introduced no syntax errors, but the old pitweb compiler produces incorrect bytecode from the merged pipeline source. This is a classic bootstrapping problem: the new pipeline needs a compatible compiler to build itself, but the only available compiler (old pitweb) miscompiles it.

Current State

• boot/tokenize.cm.mcode through boot/streamline.cm.mcode contain the old pitweb pipeline mcode (pre-merge). These pass 641/641 vm_suite tests.
• All other boot mcode files (engine, bootstrap, seed_bootstrap, plus core modules like fd, time, toml, etc.) are compiled from the merged source and work correctly.
• The merged pipeline .cm source has changes from fix_gc that are not active — the runtime uses the old pitweb pipeline mcode.

The old pitweb pipeline is NOT fully working. While it passes the test suite, it miscompiles nested function declarations. This breaks:

• toml.encode() — the encoder uses nested function declarations inside encode_toml
• Shop.save_lock() — calls toml.encode(), so any lock.toml mutation fails
• Any other .cm module that uses nested named function declarations

This means the ID-based package symbol naming (Phase 2 in the plan) is blocked: it needs save_lock() to persist package IDs to lock.toml.

The shop.cm changes for ID-based naming are already written and correct — they just need a working pipeline underneath. Once the pipeline is fixed, the ID system will work.

What Changed in the Pipeline

The fix_gc merge brought these changes to the pipeline .cm files:

• mcode.cm: Type-guarded arithmetic (emit_add_decomposed now generates is_text/is_num checks instead of letting the VM dispatch), emit_numeric_binop for subtract/multiply/etc., sensory_ops lookup table, array/record literal count args (["array", dest, count] instead of ["array", dest, 0])
• fold.cm: Lookup tables (binary_ops, unary_ops, assign_ops, etc.) replacing if-chains, combined "array" and "text literal" handling
• tokenize.cm: ~500 lines of changes
• streamline.cm: ~700 lines of changes
• parse.cm: ~40 lines of changes (minor)

Regen Flags

regen.ce now has two modes:

./cell --dev --seed regen          # default: skip pipeline files
./cell --dev --seed regen --all    # include pipeline files (tokenize/parse/fold/mcode/streamline)

The default mode is safe — it regenerates everything except the 5 pipeline files, preserving the working old pitweb pipeline mcode.

How to Fix

The goal is to get the merged pipeline .cm source to produce working mcode when compiled by the current (old pitweb) pipeline. The process:

1. Start from the current repo state (old pitweb pipeline mcode in boot/)
2. Edit one or more pipeline .cm files to fix the issue
3. Regen with --all to recompile everything including pipeline:

   ./cell --dev --seed regen --all
   

4. Test the new pipeline with a simple sanity check:

   rm -rf .cell/build/*
   echo 'var x = 42; print(x)' > /tmp/test.ce
   ./cell --dev --seed /tmp/test
   

5. If that works, run the full test suite:

   rm -rf .cell/build/*
   ./cell --dev vm_suite
   

6. If tests pass, regen again (the new pipeline compiles itself):

   ./cell --dev --seed regen --all
   

7. Repeat steps 4-6 until idempotent — two consecutive regen --all runs produce identical boot mcode and all tests pass.

Debugging Tips

• The old pitweb pipeline mcode is always available via:

  git checkout HEAD^1 -- boot/tokenize.cm.mcode boot/parse.cm.mcode \
    boot/fold.cm.mcode boot/mcode.cm.mcode boot/streamline.cm.mcode
  

• Use --seed mode for testing compilation — it bypasses the engine entirely and loads the pipeline directly from boot mcode.
• The failure mode is silent: the old compiler compiles the new source without errors but produces wrong bytecode.
• Known broken patterns with the old pitweb pipeline:
  • var x = 42; print(x) fails when compiled by the regenned pipeline mcode
  • Nested named function declarations (function foo() {} inside another function) produce "not a function" errors — this breaks toml.encode()
  • Test with: echo 'var toml = use("toml"); print(toml.encode({a: 1}))' > /tmp/t.ce && ./cell --dev /tmp/t.ce
• The most likely culprits are the mcode.cm changes (type-guarded arithmetic, array/record count args) since these change the bytecode format. The fold.cm changes (lookup tables) are more likely safe refactors.