makepad/handoff.md

# handoff — the Makepad AI Game Maker (godot-like livecoding agent)

State as of 2026-07-10. Everything below is in the working tree, **nothing committed**.
Written as the handoff for `examples/gamemaker`: a voice-driven game maker for kids
where Claude Code live-edits a single splash script and the game hot-reloads in-process
— the replacement for the Godot-backed `examples/godot` (which still exists, untouched,
as the reference implementation).

## 1. What it is

```
kid holds F1, talks (Whisper) ──► Claude Code (via makepad_ai::ClaudeCodeAgent)
        ▲                              │ edits ~/games/<name>/game.splash
        │ TTS speaks replies           ▼
Game Maker app ── chat pane │ GameView pane: splash isolate + fixed-60Hz world,
                            │ re-evals on every edit (aichat-style streaming),
                            │ broken edits never replace the running world
```

- One process. No engine restarts, no focus stealing. A clean edit is visible mid-turn
  ("the world grows while the AI talks"); a broken edit keeps the last-good world
  running and returns line-numbered errors to the agent.
- The whole game is ONE file, `game.splash`, written in a curated ~45-verb `game.*`
  DSL (deliberately smaller than GDScript — easier to prompt, verify, and keep stable).
- The agent self-verifies through `tools/ag`: screenshots, deterministic input-tape
  playtests with numeric probes, error/log round-trip.

## 2. Engine architecture (examples/gamemaker)

**Host** (`src/main.rs`, cloned from the godot example): chat/voice/TTS/session shell,
per-game state in `<game>/.gamemaker/` (chat log, Claude session id, model), project
dropdown over `~/games/*/game.splash` (`GAMEMAKER_HOME` overrides), harness re-stamped
into each game on switch (`tools/`, `CLAUDE.md` — app-owned, kid files never touched).
System prompt teaches kid-talk rules + the DSL + the always-check-errors workflow.
Permission policy: `Edit/Write(./**)` + `Bash(./tools/ag:*)` only.

**GameView** (`src/game_view.rs`, ~2k lines — splitting into modules was repeatedly
deferred, do it before it grows again):
- **Script hosting**: dedicated splash isolate (`cx.alloc_splash_vm_with_network`),
  incremental re-eval via `eval_with_append_source` parser checkpoints (the same
  machinery aichat uses to stream `runsplash` blocks), instruction limits (2M per
  eval / 500k per tick), pointer-stable ScriptMod keying. Reload triggers: 250ms mtime
  watch + immediately on the agent's Edit/Write ToolRequest completing.
- **Last-good semantics**: eval snapshots the world (entities, parts, labels, terrain,
  sky, HUD), rebuilds from scratch, rolls back wholesale on failure. Errors go to
  `.agent/game.log` + `.agent/last_error.txt` (cleared-to-empty on success BY DESIGN —
  "empty = your edit is live") via a platform addition: `ScriptVm::take_errors` +
  `captured_errors` sink (streaming evals otherwise silence errors).
- **Dispatch design**: one native `game` handle over `Rc<RefCell<GameWorld>>`; every
  `game.verb(...)` is a match arm in `game_dispatch` — synchronous mutation, no async
  widget trampoline, deterministic ordering. Adding a verb = one arm + one doc row.
- **Physics**: deliberately tiny — gravity + axis-separated AABB sweeps, kinematic
  carry, sensors, mover ground detection, terrain height-lookup collision with 0.55
  step-up. **The physics body NEVER rotates — only the visual model yaws.** This
  matches Godot exactly (their CharacterBody3D does the same); it is the reason a
  mini-engine reaches parity. box3d swap remains a marked seam (see §5).
- **Renderer**: offscreen 3D pass composited into the pane. Shaders (script-DSL,
  JIT-compiled on headless): `DrawGameCube` (instanced lit boxes + emission + fog),
  `DrawGameAlpha` (translucent pass: sensors, water, blob shadows), `DrawGameSky`
  (gradient dome), `DrawGameTerrain` (triangulated heightfield, per-vertex color).
  Near plane **1.0** (Godot's CAM_NEAR) so lens-overlapping creatures clip open
  instead of filling the screen. Billboard labels project into a 2D overlay with
  automatic 4-copy dark outlines; HUD slots + crosshair are plain overlay draws.
- **Input**: ActionMap — keyboard (WASD/arrows, Space, F=shoot, G=grab) + gamepad
  (`cx.game_input_states`: stick analog w/ deadzone, dpad, A=jump/X=shoot/B=grab,
  edge-detected) merged at read time via `PadState` (never into the key-held set, so
  devices can't cancel each other). Input snapshot per tick: held/pressed booleans,
  raw `axis_x/axis_z`, **camera-relative `move_x/move_z`** (the canonical walk input;
  the axes rotated by effective camera yaw — derived from the camera basis, and the
  fix for "controls don't match the camera": the first documented recipe rotated by
  −yaw). Tapes zero the pad and pin yaw+pitch → byte-identical replays.
- **Audio** (`src/synth.rs`): polyphonic synth (osc + percussive envelope, 24-voice
  cap) mixed into the app's audio callback under the TTS voice; mute button silences
  both. Named bank (18: jump shoot zap grab angry calm rescue shove board coin hurt
  win lose squeak roar bark moo clank whip) + `game.beep{}` + `game.jingle("C5 E5")`.
  The Godot corpus AI hand-built a synth in GDScript when it had no audio API —
  that's why this is an engine service.
- **Agent harness** (`tools/ag` + file RPC through `.agent/`): `peek` (4 live frames
  via `Cx::capture_next_frame_to_file` — a platform addition riding the studio
  screenshot pipeline — + entity state), `test N tape` (restart, frame-indexed tape
  through the ActionMap, captures + `probe.txt`, `sheet.py` contact sheet), `errors`,
  `logs`. Tape format is the Godot harness's JSON unchanged. Errors are also
  PUSHED into the agent chat (GameViewAction → auto fix wake-up on post-turn broken
  evals and idle runtime errors; 2-wake-up guard, reset by a kid message;
  GAMEMAKER_NO_AGENT=1 disables the agent for token-safe headless tests). Engine
  registration errors surface in the status bar, never in the kid chat.

**DSL surface** — the authoritative, always-in-sync docs:
- **repo-root `splashgame.md`** — THE agent-facing API contract, loaded into the
  system prompt at runtime (fs read + include_str fallback), exactly the pattern
  aichat uses for `splash.md`. Adding a verb = dispatch arm + a row here.
- `resources/template/CLAUDE.md` — per-game workflow only (ag ritual, house style,
  gotchas); points at the system prompt for the API.
- `resources/aigame-dsl.md` — developer guide (execution model, extension points).
Spawning: `box/mover/spawn/terrain/part/label`. Driving: `on_tick(dt, input)`, `walk/
jump/on_floor/pos/vel/set_pos/set_vel/face/yaw`, `find/tag/distance`, timers `after`,
`on_touch`. Look: `set_color/glow/scale/move_part/sky/camera` (orbit / `side:` /
`third_person:` with occlusion pull-in ignoring `"scenery"`). Systems: `attach`
(seat + `mode:"ride"` w/ spin)/`detach`, `speed_mult`, `beam`, `ground_y/ground_peak`,
`rand/rand_range` (seeded per eval — tape-deterministic, better than the corpus's
`randomize()`), HUD `text(slot,…)/crosshair`, labels w/ ids + `label_text`, sounds.
Terrain noise is shaped engine-side (`freq/offset/amp/step/min/max/plaza`, ≤384
cells, `bands:` height-color thresholds — snow bands are what make hills read as
mountains) so big worlds cost no script instruction budget.

## 3. How parity was reached (method matters as much as the result)

1. **Corpus first**: the AI-generated Godot game (`~/games/my-game`, grew to ~44
   creatures) was inventoried exhaustively (`aigame_port_inventory.md`), and every
   engine feature exists because the corpus used it — nothing speculative. Confirmed
   non-needs (no tweens/particles/navmesh/shaders in 2600+ generated lines) were
   deliberately NOT built.
2. **Port as acceptance test**: the game was ported to
   `resources/fixtures/sandbox3d.splash` (932 lines, 90 entities — ~2100 lines of
   GDScript; the compression is the argument the DSL is at the right altitude).
   Each port round produced findings (`aigame_port_findings.md`); findings became
   engine verbs; the fixture re-ported until clean. Gap analysis lives in
   `aigame_parity_gap.md`; the original plan in `aigame.md`.
3. **Verification discipline**: headless eval cleanliness + numeric probes against
   Godot ground truth (plaza floor exactly 7.9, walk exactly 6.0, jump apex JUMP²/2G,
   ride-debuff exactly 3.0) + determinism (two tape runs byte-identical). Pixels on
   GPU were checked by rik; pure-headless pixels are blocked on §5.5.

The fixture runs the CURRENT game: Giant DogDay guardian (intercept-charge, beam
bonk-arc), 3 headcrabs (leap→latch→speed debuff→jump to shake off), the Prototype
weeping-angel (LOS dot vs camera), Baba Chops (glow-ramp fire eyes, ram), Nightmare
Huggy (arm-reach via move_part), CatNap sleep-curl (scale), Kissy bodyguard, heal
quest, 10 farm animals (per-kind sfx pitch), trucks + passengers, grapple hand on G
(terrain yank / creature haul, beam cable), 257×257 smooth terrain at Godot's exact
constants with banded snow mountains, sky/fog/blob shadows, crosshair/hint/flash HUD.
Installed for rik as `~/games/dogday-world`.

## 4. Infrastructure built along the way (reusable beyond gamemaker)

- **Headless render/test pipeline**: `MAKEPAD=headless` builds render frames to PNG
  via CPU raster + JIT-compiled Rust shaders (`--draws=N`,
  `MAKEPAD_HEADLESS_OUT_DIR`); warm cache at `examples/splash/target`. gamemaker has
  its own `build.rs` mirroring the platform's env→cfg wiring.
- **Three JIT shader-compiler bugs fixed** (platform/script + headless preamble):
  scalar casts `u32(x)` → `as` casts; `Mat4f*Mat4f` missing in the runtime preamble;
  heterogeneous constructors splat-padding because `ShaderType::Id` args weren't
  scope-resolved. Regression stages 4k/4l in `platform/script/test/src/main.rs`.
- **`ScriptVm::take_errors` + `captured_errors` sink** (platform/script/src/vm.rs) —
  the error round-trip primitive; should be upstreamed into the `Splash` widget too
  (aichat runsplash blocks still render blank on error).
- **`Cx::capture_next_frame_to_file`** (platform/src/os/cx_shared.rs + metal path) —
  in-app window capture without the studio bridge.
- **xr on box3d**: the whole `makepad-xr` crate ported off Rapier3D (raycast vehicle
  reimplemented in `xr/src/scene/raycast_vehicle.rs`; 4 pre-existing test failures
  match the Rapier baseline exactly). box3d `body_set_mass_data` stale
  world-inverse-inertia bug found + fixed with a regression test.
- The **godot example harness** got the no-focus-steal treatment first (open -g,
  unfocusable offscreen capture window, last-good relaunch policy) — kept as-is.

## 5. Remaining work for FULL parity (ranked)

1. **Caves / overhangs** — the only dropped world feature. Heightfields can't carve.
   Options: engine `game.tunnel(a, b, r)` laying rock-slab roofs (matches the Godot
   `_carve_caves` approach — it lays roof slabs too, it does NOT boolean-carve), or
   accept boxes-as-caves authored by the AI.
2. **`game.raycast(from, dir, {mask})`** — the grapple currently probes terrain
   height only; it can't catch trees/boxes/creatures mid-flight, and ledge-probe AI
   (the corpus's no-navmesh pathing trick) can't be written. One verb unlocks both.
3. **Real shadow maps** — blob shadows ground creatures but the 21:54 Godot capture
   has directional shadows. A single-cascade sun map over the play area is enough.
4. **Camera-overlap creature fade** — near-clip 1.0 fixed the giant-polygon fill;
   inside a crowd you can still sit within a body. Fade entities whose AABB
   intersects a small camera sphere (Godot mitigates via its spring-arm feel).
5. **Headless offscreen-pass compositing** (task #9) — box3d example renders nothing
   headless; gamemaker's pane is blank in pure-headless frames (GPU capture path is
   fine). Blocks CI-grade visual verification of `ag test` sheets. Look at
   `platform/src/os/headless/event_loop.rs` pass scheduling + overlay draw lists +
   NextFrame-only apps never drawing.
6. **box3d under the verbs** — the mini-AABB physics is the deliberate seam
   (`TODO(aigame)` in game_view.rs). Swap when games need slopes-with-momentum,
   stacking, ragdolls, vehicles-with-suspension. box3d is deterministic and already
   in-tree; keep the tape guarantees.
7. **The 2D side-scroller** (`main.tscn`) was never ported — `side:` camera + boxes
   cover it in principle; port it as a second fixture to harden 2D ergonomics
   (AnimatableBody movers + Camera2D limits analogues).
8. **Engine polish debt**: terrain band colors interact with auto-shade (bands win —
   fine, but no slope shading within a band); `game.terrain` columns mode still
   spawns per-cell entities (instanced draw + height collision would retire it);
   part transforms have no rot lerp shortest-path handling; gamepad stick vertical
   sign untested on hardware (one-line flip if inverted).
9. **Perf headroom**: 90 entities × parts ≈ fine; the tick budget (500k instructions)
   fits the fixture's ~15 actors of AI — a 100-creature brawl will need either budget
   raise or engine-side steering helpers (`walk_towards`, `flee`) which the corpus's
   shared `_drive()` suggests anyway.
10. **Future (from aigame.md phase 3)**: host the scene layer on `XrNode` so the same
    game.splash runs on Quest (xr is on box3d now, and has hands/multiplayer);
    upstream the error round-trip to `Splash`; batch corpus regression harness
    (splash_preview-style: run recorded kid asks through the real CLI against the
    DSL guide, eval + tape-smoke each result).

## 6. Run / verify cheatsheet

```bash
# the app (from the makepad repo root; Whisper model resolves from CWD)
cargo run -p makepad-example-gamemaker --release
# rik's install: ~/games/dogday-world (set as .last), full fixture

# headless smoke of any game dir
MAKEPAD=headless CARGO_TARGET_DIR=examples/splash/target \
  MAKEPAD_HEADLESS_OUT_DIR=/tmp/frames GAMEMAKER_HOME=<games-root> \
  cargo run -p makepad-example-gamemaker --release -- --draws=6
# then: <game>/.agent/game.log  ("eval #N: ok, E entities"), last_error.txt empty
# AND count startup script errors: ... 2>&1 | grep -cE "^\[E\]" must be 0
# (shader/widget REGISTRATION errors print as [E] lines but do not fail the run —
#  a missed [E] once shipped a build with the terrain shader dead)

# agent-side playtest, in a game dir (app running)
./tools/ag errors | logs | peek | test 200 tools/tapes/<tape>.json
# sheet: .agent/sheet.png   probes: .agent/probe.txt   (byte-identical across runs)

# engine tests touched by this work
cargo run -p makepad-script-test --release        # shader codegen stages 4k/4l
cargo test -p makepad-box3d --release             # incl. mass-data regression
cargo test -p makepad-xr --release                # 127 pass / 4 pre-existing
```

## 7. Document index

| doc | what |
|---|---|
| `aigame.md` | the original migration plan (Godot → splash), phases + rationale |
| `aigame_port_inventory.md` | exhaustive Godot API usage of the generated corpus |
| `aigame_port_findings.md` | port findings v1+v2: API cleanups, fidelity ledgers, open gaps |
| `aigame_parity_gap.md` | the final gap matrix (current game vs engine) + fork specs |
| `examples/gamemaker/resources/aigame-dsl.md` | developer guide to the DSL + engine internals |
| `splashgame.md` (repo root) | THE agent-facing API contract, loaded into the system prompt (keep in sync!) |
| `examples/gamemaker/resources/template/CLAUDE.md` | per-game workflow stamped into each project |
| this file | orientation + what's left |