161 lines
7.6 KiB
Markdown
161 lines
7.6 KiB
Markdown
---
|
|
title: "Shop Architecture"
|
|
description: "How the shop resolves, compiles, caches, and loads modules"
|
|
weight: 35
|
|
type: "docs"
|
|
---
|
|
|
|
The shop is the module resolution and loading engine behind `use()`. It handles finding modules, compiling them, caching the results, and loading C extensions. The shop lives in `internal/shop.cm`.
|
|
|
|
## Startup Pipeline
|
|
|
|
When `pit` runs a program, three layers bootstrap in sequence:
|
|
|
|
```
|
|
bootstrap.cm → engine.cm → shop.cm → user program
|
|
```
|
|
|
|
**bootstrap.cm** loads the compiler toolchain (tokenize, parse, fold, mcode, streamline) from pre-compiled bytecode. It defines `analyze()` (source to AST) and `compile_to_blob()` (AST to binary blob). It then loads engine.cm.
|
|
|
|
**engine.cm** creates the actor runtime (`$_`), defines `use_core()` for loading core modules, and populates the environment that shop receives. It then loads shop.cm via `use_core('internal/shop')`.
|
|
|
|
**shop.cm** receives its dependencies through the module environment — `analyze`, `run_ast_fn`, `use_cache`, `shop_path`, `runtime_env`, `content_hash`, `cache_path`, and others. It defines `Shop.use()`, which is the function behind every `use()` call in user code.
|
|
|
|
## Module Resolution
|
|
|
|
When `use('path')` is called from a package context, the shop resolves the module through a multi-layer search. Both the `.cm` script file and C symbol are resolved independently, and the one with the narrowest scope wins.
|
|
|
|
### Resolution Order
|
|
|
|
For a call like `use('sprite')` from package `myapp`:
|
|
|
|
1. **Own package** — `~/.pit/packages/myapp/sprite.cm` and C symbol `js_myapp_sprite_use`
|
|
2. **Aliased dependencies** — if `myapp/pit.toml` has `renderer = "gitea.pockle.world/john/renderer"`, checks `renderer/sprite.cm` and its C symbols
|
|
3. **Core** — built-in core modules and internal C symbols
|
|
|
|
For calls without a package context (from core modules), only core is searched.
|
|
|
|
### Private Modules
|
|
|
|
Paths starting with `internal/` are private to their package:
|
|
|
|
```javascript
|
|
use('internal/helpers') // OK from within the same package
|
|
// Cannot be accessed from other packages
|
|
```
|
|
|
|
### Explicit Package Imports
|
|
|
|
Paths containing a dot in the first component are treated as explicit package references:
|
|
|
|
```javascript
|
|
use('gitea.pockle.world/john/renderer/sprite')
|
|
// Resolves directly to the renderer package's sprite.cm
|
|
```
|
|
|
|
## Compilation and Caching
|
|
|
|
Every module goes through a content-addressed caching pipeline. The cache key is the BLAKE2 hash of the source content, so changing the source automatically invalidates the cache.
|
|
|
|
### Cache Hierarchy
|
|
|
|
When loading a module, the shop checks (in order):
|
|
|
|
1. **In-memory cache** — `use_cache[key]`, checked first on every `use()` call
|
|
2. **Installed dylib** — per-file `.dylib` in `~/.pit/lib/<pkg>/<stem>.dylib`
|
|
3. **Internal symbols** — statically linked into the `pit` binary (fat builds)
|
|
4. **Installed mach** — pre-compiled bytecode in `~/.pit/lib/<pkg>/<stem>.mach`
|
|
5. **Cached bytecode** — content-addressed in `~/.pit/build/<hash>` (no extension)
|
|
6. **Cached .mcode IR** — JSON IR in `~/.pit/build/<hash>.mcode`
|
|
7. **Adjacent .mach/.mcode** — files alongside the source (e.g., `sprite.mach`)
|
|
8. **Source compilation** — full pipeline: analyze, mcode, streamline, serialize
|
|
|
|
When both a `.dylib` and `.mach` exist for the same module in `lib/`, the dylib is selected. Dylib resolution also wins over internal symbols, so a dylib in `lib/` can hot-patch a fat binary. Delete the dylib to fall back to mach or static.
|
|
|
|
Results from steps 6-8 are cached back to the content-addressed store for future loads.
|
|
|
|
### Content-Addressed Store
|
|
|
|
The build cache at `~/.pit/build/` stores ephemeral artifacts named by the BLAKE2 hash of their inputs:
|
|
|
|
```
|
|
~/.pit/build/
|
|
├── a1b2c3d4... # cached bytecode blob (no extension)
|
|
├── c9d0e1f2...mcode # cached JSON IR
|
|
└── f3a4b5c6... # compiled dylib (checked before copying to lib/)
|
|
```
|
|
|
|
This scheme provides automatic cache invalidation: when source changes, its hash changes, and the old cache entry is simply never looked up again. When building a dylib, the build cache is checked first — if a matching hash exists, it is copied to `lib/` without recompiling.
|
|
|
|
### Core Module Caching
|
|
|
|
Core modules loaded via `use_core()` in engine.cm follow the same pattern. On first startup after a fresh install, core modules are compiled from `.cm.mcode` JSON IR and cached as `.mach` blobs. Subsequent startups load from cache, skipping the JSON parse and compile steps entirely.
|
|
|
|
User scripts (`.ce` files) are also cached. The first run compiles and caches; subsequent runs with unchanged source load from cache.
|
|
|
|
## C Extension Resolution
|
|
|
|
C extensions are resolved alongside script modules. A C module is identified by a symbol name derived from the package and file name:
|
|
|
|
```
|
|
package: gitea.pockle.world/john/prosperon
|
|
file: sprite.c
|
|
symbol: js_gitea_pockle_world_john_prosperon_sprite_use
|
|
```
|
|
|
|
### C Resolution Sources
|
|
|
|
1. **Installed dylibs** — per-file dylibs in `~/.pit/lib/<pkg>/<stem>.dylib` (deterministic paths, no manifests)
|
|
2. **Internal symbols** — statically linked into the `pit` binary (fat builds)
|
|
|
|
Dylibs are checked first at each resolution scope, so an installed dylib always wins over a statically linked symbol. This enables hot-patching fat binaries by placing a dylib in `lib/`.
|
|
|
|
### Name Collisions
|
|
|
|
Having both a `.cm` script and a `.c` file with the same stem at the same scope is a **build error**. For example, `render.cm` and `render.c` in the same directory will fail. Use distinct names — e.g., `render.c` for the C implementation and `render_utils.cm` for the script wrapper.
|
|
|
|
## Environment Injection
|
|
|
|
When a module is loaded, the shop builds an `env` object that becomes the module's set of free variables. This includes:
|
|
|
|
- **Runtime functions** — `logical`, `some`, `every`, `starts_with`, `ends_with`, `is_actor`, `log`, `send`, `fallback`, `parallel`, `race`, `sequence`
|
|
- **Capability injections** — actor intrinsics like `$self`, `$delay`, `$start`, `$receiver`, `$fd`, etc.
|
|
- **`use` function** — scoped to the module's package context
|
|
|
|
The set of injected capabilities is controlled by `script_inject_for()`, which can be tuned per package or file.
|
|
|
|
## Shop Directory Layout
|
|
|
|
```
|
|
~/.pit/
|
|
├── packages/ # installed packages (directories and symlinks)
|
|
│ └── core -> ... # symlink to the ƿit core
|
|
├── lib/ # INSTALLED per-file artifacts (persistent, human-readable)
|
|
│ ├── core/
|
|
│ │ ├── fd.dylib
|
|
│ │ ├── time.mach
|
|
│ │ ├── time.dylib
|
|
│ │ └── internal/
|
|
│ │ └── os.dylib
|
|
│ └── gitea_pockle_world_john_prosperon/
|
|
│ ├── sprite.dylib
|
|
│ └── render.dylib
|
|
├── build/ # EPHEMERAL cache (safe to delete anytime)
|
|
│ ├── <hash> # cached bytecode or dylib blobs (no extension)
|
|
│ └── <hash>.mcode # cached JSON IR
|
|
├── cache/ # downloaded package zip archives
|
|
├── lock.toml # installed package versions and commit hashes
|
|
└── link.toml # local development link overrides
|
|
```
|
|
|
|
## Key Files
|
|
|
|
| File | Role |
|
|
|------|------|
|
|
| `internal/bootstrap.cm` | Loads compiler, defines `analyze()` and `compile_to_blob()` |
|
|
| `internal/engine.cm` | Actor runtime, `use_core()`, environment setup |
|
|
| `internal/shop.cm` | Module resolution, compilation, caching, C extension loading |
|
|
| `internal/os.c` | OS intrinsics: dylib ops, internal symbol lookup, embedded modules |
|
|
| `package.cm` | Package directory detection, alias resolution, file listing |
|
|
| `link.cm` | Development link management (link.toml read/write) |
|