170 lines
7.4 KiB
Markdown
170 lines
7.4 KiB
Markdown
---
|
|
title: "Shop Architecture"
|
|
description: "How the shop resolves, compiles, caches, and loads modules"
|
|
weight: 35
|
|
type: "docs"
|
|
---
|
|
|
|
The shop is the module resolution and loading engine behind `use()`. It handles finding modules, compiling them, caching the results, and loading C extensions. The shop lives in `internal/shop.cm`.
|
|
|
|
## Startup Pipeline
|
|
|
|
When `pit` runs a program, three layers bootstrap in sequence:
|
|
|
|
```
|
|
bootstrap.cm → engine.cm → shop.cm → user program
|
|
```
|
|
|
|
**bootstrap.cm** loads the compiler toolchain (tokenize, parse, fold, mcode, streamline) from pre-compiled bytecode. It defines `analyze()` (source to AST) and `compile_to_blob()` (AST to binary blob). It then loads engine.cm.
|
|
|
|
**engine.cm** creates the actor runtime (`$_`), defines `use_core()` for loading core modules, and populates the environment that shop receives. It then loads shop.cm via `use_core('internal/shop')`.
|
|
|
|
**shop.cm** receives its dependencies through the module environment — `analyze`, `run_ast_fn`, `use_cache`, `shop_path`, `runtime_env`, `content_hash`, `cache_path`, and others. It defines `Shop.use()`, which is the function behind every `use()` call in user code.
|
|
|
|
## Module Resolution
|
|
|
|
When `use('path')` is called from a package context, the shop resolves the module through a multi-layer search. Both the `.cm` script file and C symbol are resolved independently, and the one with the narrowest scope wins.
|
|
|
|
### Resolution Order
|
|
|
|
For a call like `use('sprite')` from package `myapp`:
|
|
|
|
1. **Own package** — `~/.pit/packages/myapp/sprite.cm` and C symbol `js_myapp_sprite_use`
|
|
2. **Aliased dependencies** — if `myapp/pit.toml` has `renderer = "gitea.pockle.world/john/renderer"`, checks `renderer/sprite.cm` and its C symbols
|
|
3. **Core** — built-in core modules and internal C symbols
|
|
|
|
For calls without a package context (from core modules), only core is searched.
|
|
|
|
### Private Modules
|
|
|
|
Paths starting with `internal/` are private to their package:
|
|
|
|
```javascript
|
|
use('internal/helpers') // OK from within the same package
|
|
// Cannot be accessed from other packages
|
|
```
|
|
|
|
### Explicit Package Imports
|
|
|
|
Paths containing a dot in the first component are treated as explicit package references:
|
|
|
|
```javascript
|
|
use('gitea.pockle.world/john/renderer/sprite')
|
|
// Resolves directly to the renderer package's sprite.cm
|
|
```
|
|
|
|
## Compilation and Caching
|
|
|
|
Every module goes through a content-addressed caching pipeline. The cache key is the BLAKE2 hash of the source content, so changing the source automatically invalidates the cache.
|
|
|
|
### Cache Hierarchy
|
|
|
|
When loading a module, the shop checks (in order):
|
|
|
|
1. **In-memory cache** — `use_cache[key]`, checked first on every `use()` call
|
|
2. **Installed dylib** — per-file `.dylib` in `~/.pit/lib/<pkg>/<stem>.dylib`
|
|
3. **Internal symbols** — statically linked into the `pit` binary (fat builds)
|
|
4. **Cached .mach blob** — binary bytecode in `~/.pit/build/<hash>`
|
|
5. **Cached .mcode IR** — JSON IR in `~/.pit/build/<hash>.mcode`
|
|
6. **Adjacent .mach/.mcode** — files alongside the source (e.g., `sprite.mach`)
|
|
7. **Source compilation** — full pipeline: analyze, mcode, streamline, serialize
|
|
|
|
Dylib resolution wins over internal symbols, so a dylib in `lib/` can hot-patch a fat binary. Delete the dylib to fall back to the static version.
|
|
|
|
Results from steps 5-7 are cached back to the content-addressed store for future loads.
|
|
|
|
### Content-Addressed Store
|
|
|
|
All cached artifacts live in `~/.pit/build/` named by the BLAKE2 hash of their source content:
|
|
|
|
```
|
|
~/.pit/build/
|
|
├── a1b2c3d4...mach # compiled bytecode blob
|
|
├── e5f6a7b8...mach # another compiled module
|
|
├── c9d0e1f2...mcode # cached JSON IR
|
|
└── f3a4b5c6...macos_arm64.dylib # native compiled module
|
|
```
|
|
|
|
This scheme provides automatic cache invalidation: when source changes, its hash changes, and the old cache entry is simply never looked up again.
|
|
|
|
### Core Module Caching
|
|
|
|
Core modules loaded via `use_core()` in engine.cm follow the same pattern. On first startup after a fresh install, core modules are compiled from `.cm.mcode` JSON IR and cached as `.mach` blobs. Subsequent startups load from cache, skipping the JSON parse and compile steps entirely.
|
|
|
|
User scripts (`.ce` files) are also cached. The first run compiles and caches; subsequent runs with unchanged source load from cache.
|
|
|
|
## C Extension Resolution
|
|
|
|
C extensions are resolved alongside script modules. A C module is identified by a symbol name derived from the package and file name:
|
|
|
|
```
|
|
package: gitea.pockle.world/john/prosperon
|
|
file: sprite.c
|
|
symbol: js_gitea_pockle_world_john_prosperon_sprite_use
|
|
```
|
|
|
|
### C Resolution Sources
|
|
|
|
1. **Installed dylibs** — per-file dylibs in `~/.pit/lib/<pkg>/<stem>.dylib` (deterministic paths, no manifests)
|
|
2. **Internal symbols** — statically linked into the `pit` binary (fat builds)
|
|
|
|
Dylibs are checked first at each resolution scope, so an installed dylib always wins over a statically linked symbol. This enables hot-patching fat binaries by placing a dylib in `lib/`.
|
|
|
|
### Combined Resolution
|
|
|
|
When both a `.cm` script and a C symbol exist for the same module name, both are resolved. The C module is loaded first (as the base), then the `.cm` script can extend it:
|
|
|
|
```javascript
|
|
// render.cm — extends the C render module
|
|
var c_render = use('internal/render_c')
|
|
// Add ƿit-level helpers on top of C functions
|
|
return record(c_render, {
|
|
draw_circle: function(x, y, r) { /* ... */ }
|
|
})
|
|
```
|
|
|
|
## Environment Injection
|
|
|
|
When a module is loaded, the shop builds an `env` object that becomes the module's set of free variables. This includes:
|
|
|
|
- **Runtime functions** — `logical`, `some`, `every`, `starts_with`, `ends_with`, `is_actor`, `log`, `send`, `fallback`, `parallel`, `race`, `sequence`
|
|
- **Capability injections** — actor intrinsics like `$self`, `$delay`, `$start`, `$receiver`, `$fd`, etc.
|
|
- **`use` function** — scoped to the module's package context
|
|
|
|
The set of injected capabilities is controlled by `script_inject_for()`, which can be tuned per package or file.
|
|
|
|
## Shop Directory Layout
|
|
|
|
```
|
|
~/.pit/
|
|
├── packages/ # installed packages (directories and symlinks)
|
|
│ └── core -> ... # symlink to the ƿit core
|
|
├── lib/ # INSTALLED per-file dylibs (persistent, human-readable)
|
|
│ ├── core/
|
|
│ │ ├── fd.dylib
|
|
│ │ ├── time.dylib
|
|
│ │ └── internal/
|
|
│ │ └── os.dylib
|
|
│ └── gitea_pockle_world_john_prosperon/
|
|
│ ├── sprite.dylib
|
|
│ └── render.dylib
|
|
├── build/ # EPHEMERAL cache (safe to delete anytime)
|
|
│ ├── <hash> # cached bytecode blobs
|
|
│ ├── <hash>.mcode # cached JSON IR
|
|
│ └── <hash>.o # compiled object files
|
|
├── cache/ # downloaded package zip archives
|
|
├── lock.toml # installed package versions and commit hashes
|
|
└── link.toml # local development link overrides
|
|
```
|
|
|
|
## Key Files
|
|
|
|
| File | Role |
|
|
|------|------|
|
|
| `internal/bootstrap.cm` | Loads compiler, defines `analyze()` and `compile_to_blob()` |
|
|
| `internal/engine.cm` | Actor runtime, `use_core()`, environment setup |
|
|
| `internal/shop.cm` | Module resolution, compilation, caching, C extension loading |
|
|
| `internal/os.c` | OS intrinsics: dylib ops, internal symbol lookup, embedded modules |
|
|
| `package.cm` | Package directory detection, alias resolution, file listing |
|
|
| `link.cm` | Development link management (link.toml read/write) |
|