proper shop caching

docs/_index.md

@@ -34,6 +34,7 @@ pit hello
 - [**Actors and Modules**](/docs/actors/) — the execution model
 - [**Requestors**](/docs/requestors/) — asynchronous composition
 - [**Packages**](/docs/packages/) — code organization and sharing
+- [**Shop Architecture**](/docs/shop/) — module resolution, compilation, and caching
 
 ## Reference
 

docs/shop.md

@@ -0,0 +1,169 @@
+---
+title: "Shop Architecture"
+description: "How the shop resolves, compiles, caches, and loads modules"
+weight: 35
+type: "docs"
+---
+
+The shop is the module resolution and loading engine behind `use()`. It handles finding modules, compiling them, caching the results, and loading C extensions. The shop lives in `internal/shop.cm`.
+
+## Startup Pipeline
+
+When `pit` runs a program, three layers bootstrap in sequence:
+
+```
+bootstrap.cm → engine.cm → shop.cm → user program
+```
+
+**bootstrap.cm** loads the compiler toolchain (tokenize, parse, fold, mcode, streamline) from pre-compiled bytecode. It defines `analyze()` (source to AST) and `compile_to_blob()` (AST to binary blob). It then loads engine.cm.
+
+**engine.cm** creates the actor runtime (`$_`), defines `use_core()` for loading core modules, and populates the environment that shop receives. It then loads shop.cm via `use_core('internal/shop')`.
+
+**shop.cm** receives its dependencies through the module environment — `analyze`, `run_ast_fn`, `use_cache`, `shop_path`, `runtime_env`, `content_hash`, `cache_path`, and others. It defines `Shop.use()`, which is the function behind every `use()` call in user code.
+
+## Module Resolution
+
+When `use('path')` is called from a package context, the shop resolves the module through a multi-layer search. Both the `.cm` script file and C symbol are resolved independently, and the one with the narrowest scope wins.
+
+### Resolution Order
+
+For a call like `use('sprite')` from package `myapp`:
+
+1. **Own package** — `~/.pit/packages/myapp/sprite.cm` and C symbol `js_myapp_sprite_use`
+2. **Aliased dependencies** — if `myapp/pit.toml` has `renderer = "gitea.pockle.world/john/renderer"`, checks `renderer/sprite.cm` and its C symbols
+3. **Core** — built-in core modules and internal C symbols
+
+For calls without a package context (from core modules), only core is searched.
+
+### Private Modules
+
+Paths starting with `internal/` are private to their package:
+
+```javascript
+use('internal/helpers')  // OK from within the same package
+// Cannot be accessed from other packages
+```
+
+### Explicit Package Imports
+
+Paths containing a dot in the first component are treated as explicit package references:
+
+```javascript
+use('gitea.pockle.world/john/renderer/sprite')
+// Resolves directly to the renderer package's sprite.cm
+```
+
+## Compilation and Caching
+
+Every module goes through a content-addressed caching pipeline. The cache key is the BLAKE2 hash of the source content, so changing the source automatically invalidates the cache.
+
+### Cache Hierarchy
+
+When loading a module, the shop checks (in order):
+
+1. **In-memory cache** — `use_cache[key]`, checked first on every `use()` call
+2. **Native dylib** — pre-compiled platform-specific `.dylib` in the content-addressed store
+3. **Cached .mach blob** — binary bytecode in `~/.pit/build/<hash>.mach`
+4. **Cached .mcode IR** — JSON IR in `~/.pit/build/<hash>.mcode`
+5. **Adjacent .mach/.mcode** — files alongside the source (e.g., `sprite.mach`)
+6. **Source compilation** — full pipeline: analyze, mcode, streamline, serialize
+
+Results from steps 4-6 are cached back to the content-addressed store for future loads.
+
+### Content-Addressed Store
+
+All cached artifacts live in `~/.pit/build/` named by the BLAKE2 hash of their source content:
+
+```
+~/.pit/build/
+├── a1b2c3d4...mach       # compiled bytecode blob
+├── e5f6a7b8...mach       # another compiled module
+├── c9d0e1f2...mcode      # cached JSON IR
+└── f3a4b5c6...macos_arm64.dylib  # native compiled module
+```
+
+This scheme provides automatic cache invalidation: when source changes, its hash changes, and the old cache entry is simply never looked up again.
+
+### Core Module Caching
+
+Core modules loaded via `use_core()` in engine.cm follow the same pattern. On first startup after a fresh install, core modules are compiled from `.cm.mcode` JSON IR and cached as `.mach` blobs. Subsequent startups load from cache, skipping the JSON parse and compile steps entirely.
+
+User scripts (`.ce` files) are also cached. The first run compiles and caches; subsequent runs with unchanged source load from cache.
+
+## C Extension Resolution
+
+C extensions are resolved alongside script modules. A C module is identified by a symbol name derived from the package and file name:
+
+```
+package: gitea.pockle.world/john/prosperon
+file:    sprite.c
+symbol:  js_gitea_pockle_world_john_prosperon_sprite_use
+```
+
+### C Resolution Sources
+
+1. **Internal symbols** — statically linked into the `pit` binary (core modules)
+2. **Per-module dylibs** — loaded from `~/.pit/lib/` via a manifest file
+
+### Manifest Files
+
+Each package with C extensions has a manifest at `~/.pit/lib/<package>.manifest.json` mapping symbol names to dylib paths:
+
+```json
+{
+  "js_mypackage_render_use": "/Users/john/.pit/lib/mypackage_render.dylib",
+  "js_mypackage_audio_use": "/Users/john/.pit/lib/mypackage_audio.dylib"
+}
+```
+
+The shop loads manifests lazily on first access and caches them.
+
+### Combined Resolution
+
+When both a `.cm` script and a C symbol exist for the same module name, both are resolved. The C module is loaded first (as the base), then the `.cm` script can extend it:
+
+```javascript
+// render.cm — extends the C render module
+var c_render = use('internal/render_c')
+// Add ƿit-level helpers on top of C functions
+return record(c_render, {
+  draw_circle: function(x, y, r) { /* ... */ }
+})
+```
+
+## Environment Injection
+
+When a module is loaded, the shop builds an `env` object that becomes the module's set of free variables. This includes:
+
+- **Runtime functions** — `logical`, `some`, `every`, `starts_with`, `ends_with`, `is_actor`, `log`, `send`, `fallback`, `parallel`, `race`, `sequence`
+- **Capability injections** — actor intrinsics like `$self`, `$delay`, `$start`, `$receiver`, `$fd`, etc.
+- **`use` function** — scoped to the module's package context
+
+The set of injected capabilities is controlled by `script_inject_for()`, which can be tuned per package or file.
+
+## Shop Directory Layout
+
+```
+~/.pit/
+├── packages/         # installed packages (directories and symlinks)
+│   └── core -> ...   # symlink to the ƿit core
+├── lib/              # compiled C extension dylibs + manifests
+├── build/            # content-addressed compilation cache
+│   ├── <hash>.mach   # cached bytecode blobs
+│   ├── <hash>.mcode  # cached JSON IR
+│   └── <hash>.<target>.dylib  # native compiled modules
+├── cache/            # downloaded package zip archives
+├── lock.toml         # installed package versions and commit hashes
+└── link.toml         # local development link overrides
+```
+
+## Key Files
+
+| File | Role |
+|------|------|
+| `internal/bootstrap.cm` | Loads compiler, defines `analyze()` and `compile_to_blob()` |
+| `internal/engine.cm` | Actor runtime, `use_core()`, environment setup |
+| `internal/shop.cm` | Module resolution, compilation, caching, C extension loading |
+| `internal/os.c` | OS intrinsics: dylib ops, internal symbol lookup, embedded modules |
+| `package.cm` | Package directory detection, alias resolution, file listing |
+| `link.cm` | Development link management (link.toml read/write) |

internal/bootstrap.cm

@@ -183,13 +183,24 @@ function run_ast(name, ast, env) {
     delete optimized._verify
     delete optimized._verify_mod
   }
-  return mach_eval_mcode(name, json.encode(optimized), env)
+  var mcode_json = json.encode(optimized)
+  var mach_blob = mach_compile_mcode_bin(name, mcode_json)
+  return mach_load(mach_blob, env)
 }
 
 // Run AST through mcode pipeline WITHOUT optimization → register VM
 function run_ast_noopt(name, ast, env) {
   var compiled = mcode_mod(ast)
-  return mach_eval_mcode(name, json.encode(compiled), env)
+  var mcode_json = json.encode(compiled)
+  var mach_blob = mach_compile_mcode_bin(name, mcode_json)
+  return mach_load(mach_blob, env)
+}
+
+// Compile AST to blob without loading (for caching)
+function compile_to_blob(name, ast) {
+  var compiled = mcode_mod(ast)
+  var optimized = streamline_mod(compiled)
+  return mach_compile_mcode_bin(name, json.encode(optimized))
 }
 
 // Helper to load engine.cm and run it with given env
@@ -248,7 +259,9 @@ if (args != null) {
     init: {program: program, arg: user_args},
     core_path: core_path, shop_path: shop_path, json: json,
     analyze: analyze, run_ast_fn: run_ast, run_ast_noopt_fn: run_ast_noopt,
-    use_cache: use_cache
+    use_cache: use_cache,
+    content_hash: content_hash, cache_path: cache_path,
+    ensure_build_dir: ensure_build_dir, compile_to_blob_fn: compile_to_blob
   })
 } else {
   // Actor spawn mode — load engine.cm with full actor env
@@ -256,6 +269,8 @@ if (args != null) {
     os: os, actorsym: actorsym, init: init,
     core_path: core_path, shop_path: shop_path, json: json, nota: nota, wota: wota,
     analyze: analyze, run_ast_fn: run_ast, run_ast_noopt_fn: run_ast_noopt,
-    use_cache: use_cache
+    use_cache: use_cache,
+    content_hash: content_hash, cache_path: cache_path,
+    ensure_build_dir: ensure_build_dir, compile_to_blob_fn: compile_to_blob
   })
 }

internal/engine.cm

@@ -1,4 +1,4 @@
-// Hidden vars (os, actorsym, init, core_path, shop_path, analyze, run_ast_fn, run_ast_noopt_fn, json, use_cache) come from env
+// Hidden vars (os, actorsym, init, core_path, shop_path, analyze, run_ast_fn, run_ast_noopt_fn, json, use_cache, content_hash, cache_path, ensure_build_dir, compile_to_blob_fn) come from env
 // In actor spawn mode, also: nota, wota
 var ACTORDATA = actorsym
 var SYSYM = '__SYSTEM__'
@@ -92,8 +92,25 @@ function use_core(path) {
 
   // Check for .cm.mcode JSON IR
   var mcode_path = core_path + '/' + path + '.cm.mcode'
+  var mcode_blob = null
+  var hash = null
+  var cached_path = null
+  var mach_blob = null
+  var source_blob = null
   if (fd.is_file(mcode_path)) {
-    result = mach_eval_mcode('core:' + path, text(fd.slurp(mcode_path)), env)
+    mcode_blob = fd.slurp(mcode_path)
+    hash = content_hash(mcode_blob)
+    cached_path = cache_path(hash)
+    if (cached_path && fd.is_file(cached_path)) {
+      result = mach_load(fd.slurp(cached_path), env)
+    } else {
+      mach_blob = mach_compile_mcode_bin('core:' + path, text(mcode_blob))
+      if (cached_path) {
+        ensure_build_dir()
+        fd.slurpwrite(cached_path, mach_blob)
+      }
+      result = mach_load(mach_blob, env)
+    }
     use_cache[cache_key] = result
     return result
   }
@@ -101,9 +118,21 @@ function use_core(path) {
   // Fall back to source .cm file — compile at runtime
   var file_path = core_path + '/' + path + MOD_EXT
   if (fd.is_file(file_path)) {
-    script = text(fd.slurp(file_path))
-    ast = analyze(script, file_path)
-    result = run_ast_fn('core:' + path, ast, env)
+    source_blob = fd.slurp(file_path)
+    hash = content_hash(source_blob)
+    cached_path = cache_path(hash)
+    if (cached_path && fd.is_file(cached_path)) {
+      result = mach_load(fd.slurp(cached_path), env)
+    } else {
+      script = text(source_blob)
+      ast = analyze(script, file_path)
+      mach_blob = compile_to_blob_fn('core:' + path, ast)
+      if (cached_path) {
+        ensure_build_dir()
+        fd.slurpwrite(cached_path, mach_blob)
+      }
+      result = mach_load(mach_blob, env)
+    }
     use_cache[cache_key] = result
     return result
   }
@@ -215,15 +244,27 @@ function create_actor(desc) {
 var $_ = {}
 $_.self = create_actor()
 
-os.use_cache = use_cache
-os.global_shop_path = shop_path
-os.$_ = $_
-os.analyze = analyze
-os.run_ast_fn = run_ast_fn
-os.run_ast_noopt_fn = run_ast_noopt_fn
-os.json = json
 use_cache['core/json'] = json
 
+// Create runtime_env early (empty) — filled after pronto loads.
+// Shop accesses it lazily (in inject_env, called at module-use time, not load time)
+// so it sees the filled version.
+var runtime_env = {}
+
+// Populate core_extras with everything shop (and other core modules) need
+core_extras.use_cache = use_cache
+core_extras.shop_path = shop_path
+core_extras.analyze = analyze
+core_extras.run_ast_fn = run_ast_fn
+core_extras.run_ast_noopt_fn = run_ast_noopt_fn
+core_extras.core_json = json
+core_extras.actor_api = $_
+core_extras.runtime_env = runtime_env
+core_extras.content_hash = content_hash
+core_extras.cache_path = cache_path
+core_extras.ensure_build_dir = ensure_build_dir
+
+// NOW load shop — it receives all of the above via env
 var shop = use_core('internal/shop')
 var time = use_core('time')
 
@@ -233,29 +274,24 @@ var parallel = pronto.parallel
 var race = pronto.race
 var sequence = pronto.sequence
 
-// Create runtime environment for modules
-var runtime_env = {
-  logical: logical,
-  some: some,
-  every: every,
-  starts_with: starts_with,
-  ends_with: ends_with,
-  actor: actor,
-  is_actor: is_actor,
-  log: log,
-  send: send,
-  fallback: fallback,
-  parallel: parallel,
-  race: race,
-  sequence: sequence
-}
+// Fill runtime_env (same object reference shop holds)
+runtime_env.logical = logical
+runtime_env.some = some
+runtime_env.every = every
+runtime_env.starts_with = starts_with
+runtime_env.ends_with = ends_with
+runtime_env.actor = actor
+runtime_env.is_actor = is_actor
+runtime_env.log = log
+runtime_env.send = send
+runtime_env.fallback = fallback
+runtime_env.parallel = parallel
+runtime_env.race = race
+runtime_env.sequence = sequence
 
 // Make runtime functions available to modules loaded via use_core
 arrfor(array(runtime_env), function(k) { core_extras[k] = runtime_env[k] })
 
-// Pass to os for shop to access
-os.runtime_env = runtime_env
-
 $_.time_limit = function(requestor, seconds)
 {
   if (!pronto.is_requestor(requestor)) {
@@ -890,9 +926,25 @@ $_.clock(_ => {
   env.args = _cell.args.arg
   env.log  = log
 
-  var script = text(fd.slurp(prog_path))
-  var ast = analyze(script, prog_path)
-  var val = run_ast_fn(prog, ast, env)
+  var source_blob = fd.slurp(prog_path)
+  var hash = content_hash(source_blob)
+  var cached_path = cache_path(hash)
+  var val = null
+  var script = null
+  var ast = null
+  var mach_blob = null
+  if (cached_path && fd.is_file(cached_path)) {
+    val = mach_load(fd.slurp(cached_path), env)
+  } else {
+    script = text(source_blob)
+    ast = analyze(script, prog_path)
+    mach_blob = compile_to_blob_fn(prog, ast)
+    if (cached_path) {
+      ensure_build_dir()
+      fd.slurpwrite(cached_path, mach_blob)
+    }
+    val = mach_load(mach_blob, env)
+  }
   if (val) {
     log.error('Program must not return anything')
     disrupt

internal/shop.cm

@@ -12,9 +12,12 @@ var pkg_tools = use('package')
 var os = use('os')
 var link = use('link')
 
-var analyze = os.analyze
-var run_ast_fn = os.run_ast_fn
-var shop_json = os.json
+// These come from env (via core_extras in engine.cm):
+//   analyze, run_ast_fn, core_json, use_cache, shop_path, actor_api, runtime_env,
+//   content_hash, cache_path, ensure_build_dir
+var shop_json = core_json
+var global_shop_path = shop_path
+var my$_ = actor_api
 
 var core = "core"
 
@@ -45,11 +48,6 @@ function ensure_dir(path) {
   }
 }
 
-function content_hash(content)
-{
-  return text(crypto.blake2(content), 'h')
-}
-
 function hash_path(content)
 {
   return global_shop_path + '/build' + '/' + content_hash(content)
@@ -66,9 +64,6 @@ var ACTOR_EXT = '.ce'
 
 var dylib_ext = '.dylib' // Default extension
 
-var use_cache = os.use_cache
-var global_shop_path = os.global_shop_path
-var my$_ = os.$_
 
 Shop.get_package_dir = function(name) {
   return global_shop_path + '/packages/' + name
@@ -430,9 +425,8 @@ Shop.get_script_capabilities = function(path) {
 // Matches engine.cm's approach: env properties become free variables in the module.
 function inject_env(inject) {
   var env = {}
-  var rt = my$_.os ? my$_.os.runtime_env : null
-  if (rt) {
-    arrfor(array(rt), function(k) { env[k] = rt[k] })
+  if (runtime_env) {
+    arrfor(array(runtime_env), function(k) { env[k] = runtime_env[k] })
   }
 
   // Add capability injections with $ prefix

website/data/docs_nav.yaml

@@ -9,6 +9,8 @@ sections:
         url: "/docs/requestors/"
       - title: "Packages"
         url: "/docs/packages/"
+      - title: "Shop Architecture"
+        url: "/docs/shop/"
   - title: "Reference"
     pages:
       - title: "Built-in Functions"