--- title: "Shop Architecture" description: "How the shop resolves, compiles, caches, and loads modules" weight: 35 type: "docs" --- The shop is the module resolution and loading engine behind `use()`. It handles finding modules, compiling them, caching the results, and loading C extensions. The shop lives in `internal/shop.cm`. ## Startup Pipeline When `pit` runs a program, three layers bootstrap in sequence: ``` bootstrap.cm → engine.cm → shop.cm → user program ``` **bootstrap.cm** loads the compiler toolchain (tokenize, parse, fold, mcode, streamline) from pre-compiled bytecode. It defines `analyze()` (source to AST) and `compile_to_blob()` (AST to binary blob). It then loads engine.cm. **engine.cm** creates the actor runtime (`$_`), defines `use_core()` for loading core modules, and populates the environment that shop receives. It then loads shop.cm via `use_core('internal/shop')`. **shop.cm** receives its dependencies through the module environment — `analyze`, `run_ast_fn`, `use_cache`, `shop_path`, `runtime_env`, `content_hash`, `cache_path`, and others. It defines `Shop.use()`, which is the function behind every `use()` call in user code. ## Module Resolution When `use('path')` is called from a package context, the shop resolves the module through a multi-layer search. Both the `.cm` script file and C symbol are resolved independently, and the one with the narrowest scope wins. ### Resolution Order For a call like `use('sprite')` from package `myapp`: 1. **Own package** — `~/.pit/packages/myapp/sprite.cm` and C symbol `js_myapp_sprite_use` 2. **Aliased dependencies** — if `myapp/pit.toml` has `renderer = "gitea.pockle.world/john/renderer"`, checks `renderer/sprite.cm` and its C symbols 3. **Core** — built-in core modules and internal C symbols For calls without a package context (from core modules), only core is searched. ### Private Modules Paths starting with `internal/` are private to their package: ```javascript use('internal/helpers') // OK from within the same package // Cannot be accessed from other packages ``` ### Explicit Package Imports Paths containing a dot in the first component are treated as explicit package references: ```javascript use('gitea.pockle.world/john/renderer/sprite') // Resolves directly to the renderer package's sprite.cm ``` ## Compilation and Caching Every module goes through a content-addressed caching pipeline. The cache key is the BLAKE2 hash of the source content, so changing the source automatically invalidates the cache. ### Cache Hierarchy When loading a module, the shop checks (in order): 1. **In-memory cache** — `use_cache[key]`, checked first on every `use()` call 2. **Installed dylib** — per-file `.dylib` in `~/.pit/lib//.dylib` 3. **Installed mach** — pre-compiled bytecode in `~/.pit/lib//.mach` 4. **Cached bytecode** — content-addressed in `~/.pit/build/` (no extension) 5. **Cached .mcode IR** — JSON IR in `~/.pit/build/.mcode` 6. **Internal symbols** — statically linked into the `pit` binary (fat builds) 7. **Source compilation** — full pipeline: analyze, mcode, streamline, serialize When both a `.dylib` and `.mach` exist for the same module in `lib/`, the dylib is selected. Dylib resolution also wins over internal symbols, so a dylib in `lib/` can hot-patch a fat binary. Delete the dylib to fall back to mach or static. Results from steps 5-7 are cached back to the content-addressed store for future loads. Each loading method (except the in-memory cache) can be individually enabled or disabled via `shop.toml` policy flags — see [Shop Configuration](#shop-configuration) below. ### Content-Addressed Store The build cache at `~/.pit/build/` stores ephemeral artifacts named by the BLAKE2 hash of their inputs: ``` ~/.pit/build/ ├── a1b2c3d4... # cached bytecode blob (no extension) ├── c9d0e1f2...mcode # cached JSON IR └── f3a4b5c6... # compiled dylib (checked before copying to lib/) ``` This scheme provides automatic cache invalidation: when source changes, its hash changes, and the old cache entry is simply never looked up again. When building a dylib, the build cache is checked first — if a matching hash exists, it is copied to `lib/` without recompiling. ### Core Module Caching Core modules loaded via `use_core()` in engine.cm follow the same pattern. On first startup after a fresh install, core modules are compiled from `.cm.mcode` JSON IR and cached as `.mach` blobs. Subsequent startups load from cache, skipping the JSON parse and compile steps entirely. User scripts (`.ce` files) are also cached. The first run compiles and caches; subsequent runs with unchanged source load from cache. ## C Extension Resolution C extensions are resolved alongside script modules. A C module is identified by a symbol name derived from the package and file name: ``` package: gitea.pockle.world/john/prosperon file: sprite.c symbol: js_gitea_pockle_world_john_prosperon_sprite_use ``` ### C Resolution Sources 1. **Installed dylibs** — per-file dylibs in `~/.pit/lib//.dylib` (deterministic paths, no manifests) 2. **Internal symbols** — statically linked into the `pit` binary (fat builds) Dylibs are checked first at each resolution scope, so an installed dylib always wins over a statically linked symbol. This enables hot-patching fat binaries by placing a dylib in `lib/`. ### Name Collisions Having both a `.cm` script and a `.c` file with the same stem at the same scope is a **build error**. For example, `render.cm` and `render.c` in the same directory will fail. Use distinct names — e.g., `render.c` for the C implementation and `render_utils.cm` for the script wrapper. ## Environment Injection When a module is loaded, the shop builds an `env` object that becomes the module's set of free variables. This includes: - **Runtime functions** — `logical`, `some`, `every`, `starts_with`, `ends_with`, `is_actor`, `log`, `send`, `fallback`, `parallel`, `race`, `sequence` - **Capability injections** — actor intrinsics like `$self`, `$delay`, `$start`, `$receiver`, `$fd`, etc. - **`use` function** — scoped to the module's package context The set of injected capabilities is controlled by `script_inject_for()`, which can be tuned per package or file. ## Shop Configuration The shop reads an optional `shop.toml` file from the shop root (`~/.pit/shop.toml`). This file controls which loading methods are permitted through policy flags. ### Policy Flags All flags default to `true`. Set a flag to `false` to disable that loading method. ```toml [policy] allow_dylib = true # per-file .dylib loading (requires dlopen) allow_static = true # statically linked C symbols (fat builds) allow_mach = true # pre-compiled .mach bytecode (lib/ and build cache) allow_compile = true # on-the-fly source compilation ``` ### Example Configurations **Production lockdown** — only use pre-compiled artifacts, never compile from source: ```toml [policy] allow_compile = false ``` **Pure-script mode** — bytecode only, no native code: ```toml [policy] allow_dylib = false allow_static = false ``` **No dlopen platforms** — static linking and bytecode only: ```toml [policy] allow_dylib = false ``` If `shop.toml` is missing or has no `[policy]` section, all methods are enabled (default behavior). ## Shop Directory Layout ``` ~/.pit/ ├── packages/ # installed packages (directories and symlinks) │ └── core -> ... # symlink to the ƿit core ├── lib/ # INSTALLED per-file artifacts (persistent, human-readable) │ ├── core/ │ │ ├── fd.dylib │ │ ├── time.mach │ │ ├── time.dylib │ │ └── internal/ │ │ └── os.dylib │ └── gitea_pockle_world_john_prosperon/ │ ├── sprite.dylib │ └── render.dylib ├── build/ # EPHEMERAL cache (safe to delete anytime) │ ├── # cached bytecode or dylib blobs (no extension) │ └── .mcode # cached JSON IR ├── cache/ # downloaded package zip archives ├── lock.toml # installed package versions and commit hashes ├── link.toml # local development link overrides └── shop.toml # optional shop configuration and policy flags ``` ## Key Files | File | Role | |------|------| | `internal/bootstrap.cm` | Loads compiler, defines `analyze()` and `compile_to_blob()` | | `internal/engine.cm` | Actor runtime, `use_core()`, environment setup | | `internal/shop.cm` | Module resolution, compilation, caching, C extension loading | | `internal/os.c` | OS intrinsics: dylib ops, internal symbol lookup, embedded modules | | `package.cm` | Package directory detection, alias resolution, file listing | | `link.cm` | Development link management (link.toml read/write) |