231 lines
10 KiB
Markdown
231 lines
10 KiB
Markdown
---
|
|
title: "Shop Architecture"
|
|
description: "How the shop resolves, compiles, caches, and loads modules"
|
|
weight: 35
|
|
type: "docs"
|
|
---
|
|
|
|
The shop is the module resolution and loading engine behind `use()`. It handles finding modules, compiling them, caching the results, and loading C extensions. The shop lives in `internal/shop.cm`.
|
|
|
|
## Startup Pipeline
|
|
|
|
When `pit` runs a program, startup takes one of two paths:
|
|
|
|
### Fast path (warm cache)
|
|
|
|
```
|
|
C runtime → engine.cm (from cache) → shop.cm → user program
|
|
```
|
|
|
|
The C runtime hashes the source of `internal/engine.cm` with BLAKE2 and looks up the hash in the content-addressed cache (`~/.pit/build/<hash>`). On a cache hit, engine.cm loads directly — no bootstrap involved.
|
|
|
|
### Cold path (first run or cache cleared)
|
|
|
|
```
|
|
C runtime → bootstrap.cm → (seeds cache) → engine.cm (from cache) → shop.cm → user program
|
|
```
|
|
|
|
On a cache miss, the C runtime loads `boot/bootstrap.cm.mcode` (a pre-compiled seed). Bootstrap compiles engine.cm and the pipeline modules (tokenize, parse, fold, mcode, streamline) from source and caches the results. The C runtime then retries the engine cache lookup, which now succeeds.
|
|
|
|
### Engine
|
|
|
|
**engine.cm** is self-sufficient. It loads its own compilation pipeline from the content-addressed cache, with fallback to the pre-compiled seeds in `boot/`. It defines `analyze()` (source to AST), `compile_to_blob()` (AST to binary blob), and `use_core()` for loading core modules. It creates the actor runtime and loads shop.cm via `use_core('internal/shop')`.
|
|
|
|
### Shop
|
|
|
|
**shop.cm** receives its dependencies through the module environment — `analyze`, `run_ast_fn`, `use_cache`, `shop_path`, `runtime_env`, `content_hash`, `cache_path`, and others. It defines `Shop.use()`, which is the function behind every `use()` call in user code.
|
|
|
|
### Cache invalidation
|
|
|
|
Caching is content-addressed by BLAKE2 over the relevant inputs for each artifact.
|
|
|
|
- Mach/script cache keys are source-content based.
|
|
- Native (`.dylib`) cache keys include source, host target, native mode marker, native cache version, and sanitize flags.
|
|
|
|
When inputs change, the old cache entry is simply never looked up again. To force a full rebuild, delete `~/.pit/build/` (or run `cell --dev clean shop --build` in a dev workspace).
|
|
|
|
## Module Resolution
|
|
|
|
When `use('path')` is called from a package context, the shop resolves the module through a multi-layer search. Both the `.cm` script file and C symbol are resolved independently, and the one with the narrowest scope wins.
|
|
|
|
### Resolution Order
|
|
|
|
For a call like `use('sprite')` from package `myapp`:
|
|
|
|
1. **Own package** — `~/.pit/packages/myapp/sprite.cm` and C symbol `js_myapp_sprite_use`
|
|
2. **Aliased dependencies** — if `myapp/cell.toml` has `renderer = "gitea.pockle.world/john/renderer"`, checks `renderer/sprite.cm` and its C symbols
|
|
3. **Core** — built-in core modules and internal C symbols
|
|
|
|
For calls without a package context (from core modules), only core is searched.
|
|
|
|
### Private Modules
|
|
|
|
Paths starting with `internal/` are private to their package:
|
|
|
|
```javascript
|
|
use('internal/helpers') // OK from within the same package
|
|
// Cannot be accessed from other packages
|
|
```
|
|
|
|
### Explicit Package Imports
|
|
|
|
Paths containing a dot in the first component are treated as explicit package references:
|
|
|
|
```javascript
|
|
use('gitea.pockle.world/john/renderer/sprite')
|
|
// Resolves directly to the renderer package's sprite.cm
|
|
```
|
|
|
|
## Compilation and Caching
|
|
|
|
Every module goes through a content-addressed caching pipeline. Cache keys are based on the inputs that affect the output artifact, so changing any relevant input automatically invalidates the cache.
|
|
|
|
### Cache Hierarchy
|
|
|
|
When loading a module, the shop checks (in order):
|
|
|
|
1. **In-memory cache** — `use_cache[key]`, checked first on every `use()` call
|
|
2. **Installed dylib** — per-file `.dylib` in `~/.pit/lib/<pkg>/<stem>.dylib`
|
|
3. **Installed mach** — pre-compiled bytecode in `~/.pit/lib/<pkg>/<stem>.mach`
|
|
4. **Cached bytecode** — content-addressed in `~/.pit/build/<hash>` (no extension)
|
|
5. **Cached .mcode IR** — JSON IR in `~/.pit/build/<hash>.mcode`
|
|
6. **Internal symbols** — statically linked into the `pit` binary (fat builds)
|
|
7. **Source compilation** — full pipeline: analyze, mcode, streamline, serialize
|
|
|
|
When both a `.dylib` and `.mach` exist for the same module in `lib/`, the dylib is selected. Dylib resolution also wins over internal symbols, so a dylib in `lib/` can hot-patch a fat binary. Delete the dylib to fall back to mach or static.
|
|
|
|
Results from steps 5-7 are cached back to the content-addressed store for future loads.
|
|
|
|
Each loading method (except the in-memory cache) can be individually enabled or disabled via `shop.toml` policy flags — see [Shop Configuration](#shop-configuration) below.
|
|
|
|
### Content-Addressed Store
|
|
|
|
The build cache at `~/.pit/build/` stores ephemeral artifacts named by the BLAKE2 hash of their inputs:
|
|
|
|
```
|
|
~/.pit/build/
|
|
├── a1b2c3d4... # cached bytecode blob (no extension)
|
|
├── c9d0e1f2...mcode # cached JSON IR
|
|
└── f3a4b5c6... # compiled dylib (checked before copying to lib/)
|
|
```
|
|
|
|
This scheme provides automatic cache invalidation: when an input changes, its hash changes, and the old cache entry is simply never looked up again. For native dylibs, inputs include target and native cache version in addition to source.
|
|
|
|
When native codegen/runtime ABI changes, bump `NATIVE_CACHE_VERSION` in both `build.cm` and `internal/shop.cm` so stale native artifacts are never reused.
|
|
|
|
### Core Module Caching
|
|
|
|
Core modules loaded via `use_core()` in engine.cm follow the same content-addressed pattern. On first use, a module is compiled from source and cached by the BLAKE2 hash of its source content. Subsequent loads with unchanged source hit the cache directly.
|
|
|
|
User scripts (`.ce` files) are also cached. The first run compiles and caches; subsequent runs with unchanged source load from cache.
|
|
|
|
## C Extension Resolution
|
|
|
|
C extensions are resolved alongside script modules. A C module is identified by a symbol name derived from the package and file name:
|
|
|
|
```
|
|
package: gitea.pockle.world/john/prosperon
|
|
file: sprite.c
|
|
symbol: js_gitea_pockle_world_john_prosperon_sprite_use
|
|
```
|
|
|
|
### C Resolution Sources
|
|
|
|
1. **Installed dylibs** — per-file dylibs in `~/.pit/lib/<pkg>/<stem>.dylib` (deterministic paths, no manifests)
|
|
2. **Internal symbols** — statically linked into the `pit` binary (fat builds)
|
|
|
|
Dylibs are checked first at each resolution scope, so an installed dylib always wins over a statically linked symbol. This enables hot-patching fat binaries by placing a dylib in `lib/`.
|
|
|
|
### Name Collisions
|
|
|
|
Having both a `.cm` script and a `.c` file with the same stem at the same scope is a **build error**. For example, `render.cm` and `render.c` in the same directory will fail. Use distinct names — e.g., `render.c` for the C implementation and `render_utils.cm` for the script wrapper.
|
|
|
|
## Environment Injection
|
|
|
|
When a module is loaded, the shop builds an `env` object that becomes the module's set of free variables. This includes:
|
|
|
|
- **Runtime functions** — `logical`, `some`, `every`, `starts_with`, `ends_with`, `is_actor`, `log`, `send`, `fallback`, `parallel`, `race`, `sequence`
|
|
- **Capability injections** — actor intrinsics like `$self`, `$delay`, `$start`, `$receiver`, `$fd`, etc.
|
|
- **`use` function** — scoped to the module's package context
|
|
|
|
The set of injected capabilities is controlled by `script_inject_for()`, which can be tuned per package or file.
|
|
|
|
## Shop Configuration
|
|
|
|
The shop reads an optional `shop.toml` file from the shop root (`~/.pit/shop.toml`). This file controls which loading methods are permitted through policy flags.
|
|
|
|
### Policy Flags
|
|
|
|
All flags default to `true`. Set a flag to `false` to disable that loading method.
|
|
|
|
```toml
|
|
[policy]
|
|
allow_dylib = true # per-file .dylib loading (requires dlopen)
|
|
allow_static = true # statically linked C symbols (fat builds)
|
|
allow_mach = true # pre-compiled .mach bytecode (lib/ and build cache)
|
|
allow_compile = true # on-the-fly source compilation
|
|
```
|
|
|
|
### Example Configurations
|
|
|
|
**Production lockdown** — only use pre-compiled artifacts, never compile from source:
|
|
|
|
```toml
|
|
[policy]
|
|
allow_compile = false
|
|
```
|
|
|
|
**Pure-script mode** — bytecode only, no native code:
|
|
|
|
```toml
|
|
[policy]
|
|
allow_dylib = false
|
|
allow_static = false
|
|
```
|
|
|
|
**No dlopen platforms** — static linking and bytecode only:
|
|
|
|
```toml
|
|
[policy]
|
|
allow_dylib = false
|
|
```
|
|
|
|
If `shop.toml` is missing or has no `[policy]` section, all methods are enabled (default behavior).
|
|
|
|
## Shop Directory Layout
|
|
|
|
```
|
|
~/.pit/
|
|
├── packages/ # installed packages (directories and symlinks)
|
|
│ └── core -> ... # symlink to the ƿit core
|
|
├── lib/ # INSTALLED per-file artifacts (persistent, human-readable)
|
|
│ ├── core/
|
|
│ │ ├── fd.dylib
|
|
│ │ ├── time.mach
|
|
│ │ ├── time.dylib
|
|
│ │ └── internal/
|
|
│ │ └── os.dylib
|
|
│ └── gitea_pockle_world_john_prosperon/
|
|
│ ├── sprite.dylib
|
|
│ └── render.dylib
|
|
├── build/ # EPHEMERAL cache (safe to delete anytime)
|
|
│ ├── <hash> # cached bytecode or dylib blobs (no extension)
|
|
│ └── <hash>.mcode # cached JSON IR
|
|
├── cache/ # downloaded package zip archives
|
|
├── lock.toml # installed package versions and commit hashes
|
|
├── link.toml # local development link overrides
|
|
└── shop.toml # optional shop configuration and policy flags
|
|
```
|
|
|
|
## Key Files
|
|
|
|
| File | Role |
|
|
|------|------|
|
|
| `internal/bootstrap.cm` | Minimal cache seeder (cold start only) |
|
|
| `internal/engine.cm` | Self-sufficient entry point: compilation pipeline, actor runtime, `use_core()` |
|
|
| `internal/shop.cm` | Module resolution, compilation, caching, C extension loading |
|
|
| `internal/os.c` | OS intrinsics: dylib ops, internal symbol lookup, embedded modules |
|
|
| `package.cm` | Package directory detection, alias resolution, file listing |
|
|
| `link.cm` | Development link management (link.toml read/write) |
|
|
| `boot/*.cm.mcode` | Pre-compiled pipeline seeds (tokenize, parse, fold, mcode, bootstrap) |
|