cell/docs/shop.md

title, description, weight, type

title	description	weight	type
Shop Architecture	How the shop resolves, compiles, caches, and loads modules	35	docs

The shop is the module resolution and loading engine behind use(). It handles finding modules, compiling them, caching the results, and loading C extensions. The shop lives in internal/shop.cm.

Startup Pipeline

When pit runs a program, startup takes one of two paths:

Fast path (warm cache)

C runtime → engine.cm (from cache) → shop.cm → user program

The C runtime hashes the source of internal/engine.cm with BLAKE2 and looks up the hash in the content-addressed cache (~/.cell/build/<hash>). On a cache hit, engine.cm loads directly — no bootstrap involved.

Cold path (first run or cache cleared)

C runtime → bootstrap.cm → (seeds cache) → engine.cm (from cache) → shop.cm → user program

On a cache miss, the C runtime loads boot/bootstrap.cm.mcode (a pre-compiled seed). Bootstrap compiles engine.cm and the pipeline modules (tokenize, parse, fold, mcode, streamline) from source and caches the results. The C runtime then retries the engine cache lookup, which now succeeds.

Engine

engine.cm is self-sufficient. It loads its own compilation pipeline from the content-addressed cache, with fallback to the pre-compiled seeds in boot/. It defines analyze() (source to AST), compile_to_blob() (AST to binary blob), and use_core() for loading core modules. It creates the actor runtime and loads shop.cm via use_core('internal/shop').

Shop

shop.cm receives its dependencies through the module environment — analyze, run_ast_fn, use_cache, shop_path, runtime_env, content_hash, cache_path, and others. It defines Shop.use(), which is the function behind every use() call in user code.

Cache invalidation

All caching is content-addressed by BLAKE2 hash of the source. When any source file changes, its hash changes and the old cache entry is simply never looked up again. No manual invalidation is needed. To force a full rebuild, delete ~/.cell/build/.

Module Resolution

When use('path') is called from a package context, the shop resolves the module through a multi-layer search. Both the .cm script file and C symbol are resolved independently, and the one with the narrowest scope wins.

Resolution Order

For a call like use('sprite') from package myapp:

1. Own package — ~/.cell/packages/myapp/sprite.cm and C symbol js_myapp_sprite_use
2. Aliased dependencies — if myapp/cell.toml has renderer = "gitea.pockle.world/john/renderer", checks renderer/sprite.cm and its C symbols
3. Core — built-in core modules and internal C symbols

For calls without a package context (from core modules), only core is searched.

Private Modules

Paths starting with internal/ are private to their package:

use('internal/helpers')  // OK from within the same package
// Cannot be accessed from other packages

Explicit Package Imports

Paths containing a dot in the first component are treated as explicit package references:

use('gitea.pockle.world/john/renderer/sprite')
// Resolves directly to the renderer package's sprite.cm

Compilation and Caching

Every module goes through a content-addressed caching pipeline. Cache keys are based on the inputs that affect the output artifact, so changing any relevant input automatically invalidates the cache.

Cache Hierarchy

When loading a module, the shop checks (in order):

1. In-memory cache — use_cache[key], checked first on every use() call
2. Build-cache dylib — content-addressed .dylib in ~/.cell/build/<hash>, found via manifest (see Dylib Manifests)
3. Cached bytecode — content-addressed in ~/.cell/build/<hash> (no extension)
4. Internal symbols — statically linked into the cell binary (fat builds)
5. Source compilation — full pipeline: analyze, mcode, streamline, serialize

Dylib resolution wins over internal symbols, so a built dylib can hot-patch a fat binary. Results from compilation are cached back to the content-addressed store for future loads.

Each loading method (except the in-memory cache) can be individually enabled or disabled via shop.toml policy flags — see Shop Configuration below.

Content-Addressed Store

The build cache at ~/.cell/build/ stores all compiled artifacts named by the BLAKE2 hash of their inputs:

~/.cell/build/
├── a1b2c3d4...          # cached bytecode blob, object file, dylib, or manifest (no extension)
└── ...

Every artifact type uses a unique salt appended to the content before hashing, so collisions between different artifact types are impossible:

Salt	Artifact
obj	compiled C object file
dylib	linked dynamic library
native	native-compiled .cm dylib
mach	mach bytecode blob
mcode	mcode IR (JSON)
deps	cached cc -MM dependency list
fail	cached compilation failure marker
manifest	package dylib manifest (JSON)

This scheme provides automatic cache invalidation: when source changes, its hash changes, and the old cache entry is simply never looked up again.

Failure Caching

When a C file fails to compile (missing SDK headers, syntax errors, etc.), the build system writes a failure marker to the cache using the fail salt. On subsequent builds, the failure marker is found and the file is skipped immediately — no time wasted retrying files that can't compile. The failure marker is keyed on the same content as the compilation (command string + source content), so if the source changes or compiler flags change, the failure is automatically invalidated and compilation is retried.

Dylib Manifests

Dylibs live at content-addressed paths (~/.cell/build/<hash>) that can only be computed by running the full build pipeline. To allow the runtime to find pre-built dylibs without invoking the build module, cell build writes a manifest for each package. The manifest is a JSON file mapping each C module to its {file, symbol, dylib} entry. The manifest path is itself content-addressed (BLAKE2 hash of the package name + manifest salt), so the runtime can compute it from the package name alone.

At runtime, when use() needs a C module from another package, the shop reads the manifest to find the dylib path. This means cell build must be run before C modules from packages can be loaded.

For native .cm dylibs, the cache content includes source, target, native mode marker, and sanitize flags, then uses the native salt. Changing any of those inputs produces a new cache path automatically.

Core Module Caching

Core modules loaded via use_core() in engine.cm follow the same content-addressed pattern. On first use, a module is compiled from source and cached by the BLAKE2 hash of its source content. Subsequent loads with unchanged source hit the cache directly.

User scripts (.ce files) are also cached. The first run compiles and caches; subsequent runs with unchanged source load from cache.

C Extension Resolution

C extensions are resolved alongside script modules. A C module is identified by a symbol name derived from the package and file name:

package: gitea.pockle.world/john/prosperon
file:    sprite.c
symbol:  js_gitea_pockle_world_john_prosperon_sprite_use

C Resolution Sources

1. Build-cache dylibs — content-addressed dylibs in ~/.cell/build/<hash>, found via per-package manifests written by cell build
2. Internal symbols — statically linked into the cell binary (fat builds)

Dylibs are checked first at each resolution scope, so a built dylib always wins over a statically linked symbol. This enables hot-patching fat binaries.

Name Collisions

Having both a .cm script and a .c file with the same stem at the same scope is a build error. For example, render.cm and render.c in the same directory will fail. Use distinct names — e.g., render.c for the C implementation and render_utils.cm for the script wrapper.

Environment Injection

When a module is loaded, the shop builds an env object that becomes the module's set of free variables. This includes:

• Runtime functions — logical, some, every, starts_with, ends_with, is_actor, log, send, fallback, parallel, race, sequence
• Capability injections — actor intrinsics like $self, $delay, $start, $receiver, $fd, etc.
• use function — scoped to the module's package context

The set of injected capabilities is controlled by script_inject_for(), which can be tuned per package or file.

Shop Configuration

The shop reads an optional shop.toml file from the shop root (~/.cell/shop.toml). This file controls which loading methods are permitted through policy flags.

Policy Flags

All flags default to true. Set a flag to false to disable that loading method.

[policy]
allow_dylib = true     # per-file .dylib loading (requires dlopen)
allow_static = true    # statically linked C symbols (fat builds)
allow_mach = true      # pre-compiled .mach bytecode (lib/ and build cache)
allow_compile = true   # on-the-fly source compilation

Example Configurations

Production lockdown — only use pre-compiled artifacts, never compile from source:

[policy]
allow_compile = false

Pure-script mode — bytecode only, no native code:

[policy]
allow_dylib = false
allow_static = false

No dlopen platforms — static linking and bytecode only:

[policy]
allow_dylib = false

If shop.toml is missing or has no [policy] section, all methods are enabled (default behavior).

Shop Directory Layout

~/.cell/
├── packages/         # installed packages (directories and symlinks)
│   └── core -> ...   # symlink to the ƿit core
├── build/            # content-addressed cache (safe to delete anytime)
│   ├── <hash>        # cached bytecode, object file, dylib, or manifest
│   └── ...
├── cache/            # downloaded package zip archives
├── lock.toml         # installed package versions and commit hashes
├── link.toml         # local development link overrides
└── shop.toml         # optional shop configuration and policy flags

Key Files

File	Role
internal/bootstrap.cm	Minimal cache seeder (cold start only)
internal/engine.cm	Self-sufficient entry point: compilation pipeline, actor runtime, use_core()
internal/shop.cm	Module resolution, compilation, caching, C extension loading
internal/os.c	OS intrinsics: dylib ops, internal symbol lookup, embedded modules
package.cm	Package directory detection, alias resolution, file listing
link.cm	Development link management (link.toml read/write)
boot/*.cm.mcode	Pre-compiled pipeline seeds (tokenize, parse, fold, mcode, streamline, bootstrap)