makepad/docs/agents/app-remote.md

# App remote control

Use this reference when inspecting or driving a Makepad app. Ownership,
focus, and capture policy live in [AGENTS.md](../../AGENTS.md).

The app side is [platform/src/remote_core.rs](../../platform/src/remote_core.rs)
(the `COMMANDS` table); the client is [libs/remote](../../libs/remote). Run
`help` against your running instance for its current command list; it can
contain newer diagnostics than this guide.

## Launch and discovery

Build the package and the client in release mode, then launch the resulting
standalone binary with `--remote`. For a hidden verification run from the
repo root:

```sh
cargo build --release -p makepad-example-splash -p makepad-remote
MAKEPAD_HIDE_WINDOWS=1 ./target/release/makepad-example-splash --remote
```

Capture the launch output using your process tool. The startup line contains
everything needed to address and clean up that instance:

```text
[makepad-remote] pid=9931 app=makepad-example-splash socket=/tmp/makepad-remote/9931.sock
```

`--remote` (no value) or `MAKEPAD_REMOTE=1` opens a Unix-socket endpoint. No
TCP port exists and nothing listens on the network. `--remote=PORT`,
`--remote HOST:PORT` and `MAKEPAD_REMOTE=<port>` are refused (exit 2); other
machines are reached through the hub. Prefer the explicit launch flag for
agent inspection. On macOS `--focus` brings the app to the front as its
window opens; a binary launched from a terminal otherwise stays behind the
launcher.

Address the instance with the client:

```sh
P=9931   # the pid from your own launch's startup line
./target/release/makepad-remote app $P status
```

`makepad-remote app <pid|app-name|socket> <command> [k=v ...]`; `makepad
inspect <pid> <command>` speaks the same commands. The exit status is 1 when
the answer has an error line and 2 when the app is unreachable.

Do not discover an instance by taking over another running one. Retain your
own launch's PID and socket. On code changes, close that instance, rebuild,
and relaunch before inspecting again.

## Common commands

Each request is one command; the answer is the same JSON as before, one line
per answer. Tree dumps are text lines. Errors are an `{"err":...}` line.
Query parameters are `k=v` arguments with the same names; a value with spaces
is a quoted shell argument. There is no HTTP, no status code and no POST body.

| Command | Purpose |
|---|---|
| `help` | The command list as JSON lines |
| `status [w=ID]` | App/PID and window geometry; omit `w` to list windows |
| `grab [w=ID] [scale=0.5] [out=FILE]`, `gseq [n=8] [every_ms=50] [scale=0.5] [out=DIR]` | Capture the next presented frame, or N frames on request cadence (1-64, >=8 ms, <=60 s span); the app's `frames` service answers and the client writes the PNG files; prints JSON with `png` path(s) and per-frame `capture_ms`/`encode_ms` (`gseq.frames`); input `wait=1` remains a next-frame barrier |
| `gq [scale=0.5]` | Grab windows, then quit; returns `png` paths and `quit:1` |
| `snap [q=TEXT] [w=ID] [all=1]` | Widget ids/types/text and window-local rectangles |
| `dump [paths=1]` | Indented widget tree (plain lines) |
| `mouse k=click x=X y=Y` | Mouse input; kinds also include move, down, up, scroll |
| `click x=X y=Y` | Click alias |
| `key k=press c=KeyA` | Key input; down/up are also supported |
| `text t=TEXT`, `key t=TEXT` | Text/IME input |
| `drop path=ABSOLUTE_PATH x=X y=Y` | One file through native drag, drop, and drag-end events |
| `log [n=50] [since=N]` | App log tail; `n` in the reply is the latest sequence |
| `step [frames=K] [fps=60]` | Run K virtual-clock frames (needs `--virtual-clock`), see below |
| `settled` | Whether anything is still pending: `{"settled":bool,"reasons":[...]}` |
| `cap start path=ABS.mp4 [fps=60]`, `cap stop` | In-process mp4 capture of the window, see below |
| `cursor [show=1] [style=arrow]` | Synthetic cursor drawn into the frame |
| `close [w=ID]`, `resize w=W h=H`, `window k=...` | Window operations |
| `activity` | Native user activity counters, see below |
| `trace`, `midi`, `ai`, `ai transcript`, `handsoff`, `shader consts`, `shader const` | Diagnostics and overlays |
| `tweak ...`, `design <op> ...` | Design feedback and source edits, see [Tweaker](tweaker.md) |
| `quit` | Graceful shutdown without a final grab |

Images: `grab`, `gseq`, `gq` and `tweak grab` read the app's `frames`
service and the client writes the PNG files, by default to
`$TMPDIR/makepad-remote/<app>-<pid>/grab-w<window>-<seq>.png` on the client
machine (newest 64 per window kept). The command prints the JSON with the
`png` paths; `out=FILE` (grab) or `out=DIR` (gseq) chooses the location.
Metal/raw readbacks downsample before worker PNG encoding. There is no raw
PNG-to-stdout mode.

`snap` entries include `window_id` and `enabled`, plus `selected` when the
widget exposes a selection, alongside the compact `i`/`ty`/`r`/`w` fields.
When the app's widgets layer registers id paths (`Cx::widget_paths_callback`)
each entry also carries `p`, the dotted id path from the window root (named
widgets only, e.g. `main.new_note`), and `snap q=path:main.new_note` (or
`snap path=main.new_note`) matches a path exactly or as a suffix of whole
segments: `new_note` finds `main.new_note`, `note` does not (paths match
case-sensitively; the plain `q=` search does not). `dump paths=1`
lists every visible widget as `path type x y w h`.

Arguments are optional unless needed for the operation. Input commands
accept `w=ID` to target a window and `wait=1` to answer after the next frame.
With no window specified, commands generally use the first window.

Mouse buttons: `b=0` left, `b=1` right, `b=2` middle. Scroll takes `dx`
and `dy`. Add `hw=1` when testing the hardware pointer-lock/pin transform.
Keyboard modifiers are `shift=1`, `ctrl=1`, `alt=1`, and `cmd=1`.

The input parser also accepts long names such as `window`, `kind`, `button`,
`text`, and `code`. Quote values with spaces as ordinary shell arguments.

File drops require an absolute path (at most 4096 bytes) and finite,
nonnegative `x`/`y` inside the window. Optional parameters are `w`/`window`
and `wait=1`; duplicate or unknown parameters are rejected. The remote
thread sends only the path; the app owns validation and loading. The reply
reports `drop_handled` and `drag_response`, which confirm event handling,
not that an asynchronous file import has completed. For example:

```sh
makepad-remote app $P drop 'path=/absolute/path/reference car.png' x=300 y=400 wait=1
```

## Drive an owned instance

With `P` set to the pid your app printed:

```sh
makepad-remote app $P help
makepad-remote app $P snap q=press_demo
```

Read the returned `r: [x, y, width, height]`, calculate its center, and feed
that point to `click x=.. y=.. wait=1`. Do not reuse coordinates from another
window or an earlier layout.

```sh
makepad-remote app $P log n=20
makepad-remote app $P gq scale=0.5
```

Read a returned PNG with the local image viewer when visual inspection is
needed. Confirm the owned process exits after cleanup. Use `quit` if a
backend cannot grab; do not replace a failed grab with an OS screenshot.

## Deterministic runs and in-process capture

A scripted demo or a pixel test that must come out the same every time pins
the four inputs that otherwise vary between runs, at launch:

```sh
MAKEPAD_HIDE_WINDOWS=1 SANDBOX_MUTE=1 ./target/release/APP --remote \
  --virtual-clock --window 360x640@3 --seed 1
```

- `--virtual-clock` (or `MAKEPAD_VIRTUAL_CLOCK=1`): the app's time
  (`seconds_since_app_start`, `Cx::time_now`, NextFrame and Draw stamps,
  pass uniforms, `start_timeout`/`start_interval` timers, and with them
  animators and the caret blink) starts at 0 and moves only through
  `step frames=K [fps=60]`, which runs exactly K frames of `1/fps` each:
  due timers, then NextFrame, then the draw, then the present. It answers
  `{"frame":N,"time":T}` after the K-th frame is presented (and captured,
  with `"captured":M`, while a capture runs). Input and grabs between steps
  see the app at the current time; nothing advances on its own. `Cx::time_now`
  reads 2026-01-01T00:00:00Z plus the virtual time. Without the flag nothing
  changes; `step` then refuses. The flag needs `--remote` (only `step`
  moves the clock); without it, it is ignored with a log line. Children an
  app spawns inherit `MAKEPAD_VIRTUAL_CLOCK`/`MAKEPAD_SEED`; prefer the flags.
  Timers fire once per frame at most: one a handler starts or re-arms fires
  on the next frame. Two injected clicks with no step between them carry
  the same timestamp (a double click); step the frames a person would take.
  OS input and a few widgets that read the wall clock directly are not on
  the virtual clock.
  Every `step` answers within a bound: a frame that cannot be presented,
  or a capture encoder that falls behind or loses frames, fails the step
  with an error after 20 s; a failed capture is reported once and later
  steps go on uncaptured.
- `--window WxH@scale`: the first window's logical size and dpi scale,
  applied before its first frame (hidden windows too, and not fitted to the
  displays). `status` then reports `sz:[W,H]` and `px:[W*scale,H*scale]`. The
  first window applied claims it. Off macOS the native size assumes a display
  at scale 1, so check `status` there; and only macOS stops its paint beat for a
  NextFrame waiting on `step` (other backends keep ticking, harmlessly).
- `--seed N` (or `MAKEPAD_SEED=N`): Splash `random`/`random_u32` start from
  N instead of the wall clock.
- `settled` answers `{"settled":bool,"reasons":[...]}`: `next_frame` (an
  animator or anything else waiting on a frame), `redraw`, `repaint`,
  `timer` (a virtual timer due within the next frame), `step` (one is
  running), `shaders` (Metal pipelines still compiling), `loads`
  (asynchronous work whose result will change a frame, counted by kind in
  `"loads":{"image":2,...}`: image decodes and fetches, glyph rasters,
  Splash resources; subsystems and apps add kinds with
  `Cx::register_async_load_probe`). Step until it settles rather than
  sleeping.
- Loads run on workers and the network, off the virtual clock, so under
  load a picture can land a frame earlier or later from run to run.
  `step frames=K wait_loads=1` opens no frame while loads are in flight
  (each wait bounded at 30 s; then the step fails naming them, e.g.
  `image=1`), which makes such runs deterministic.
- `cap start path=/ABS/file.mp4 [fps=60] [audio=1] [overwrite=1] [w=ID]`
  starts an in-process H.264 recording of the window at native pixels. The
  directory must exist and an existing file needs `overwrite=1`; both are
  checked before the capture starts. `cap stop`
  answers once the file is finalized, with `frames`, `sz`, `missing` and a
  `hash` of the raw frames (`hashes=1` adds one per frame). Under the virtual
  clock exactly one frame per `step` frame is written, at `pts = n / fps`
  counted from the first captured step; a `step` at another fps than the
  capture's is refused. Frames presented for input or grabs between steps
  are not written. Without the virtual clock, every presented frame is
  written at its wall time, and frames the encoder cannot keep up with are
  dropped (counted in `missing`). `audio=1` adds the app's own output (the
  audio tap) on the frames' sample clock, padded with silence; the device
  runs in real time, so under the virtual clock only the last 0.1 s the
  device played is kept per frame and the sound is not deterministic (run
  with `SANDBOX_MUTE=1`). Quitting, or the app closing, with a capture open
  finishes the file. `cap pause` and `cap resume` stop and restart
  writing without stopping the clock: while paused, `step` frames advance
  the app but are not written (nor is sound), and the file stays gapless, as
  pts count written frames (`{"ok":1,"paused":1,"frames":N}` /
  `{"ok":1,"resumed":1,"frames":N}`, N written so far; an error without a
  capture). In wall-clock mode the paused time comes off the file's clock.
  The red hands-off frame and ScreenCap's REC dot are
  never in the file; the tweaker overlay is, if it is on.
- `cursor show=1 [style=arrow|hand|text|crosshair|app]` draws a synthetic
  pointer into the window's frame (so grabs and captures show it), moved by
  every injected mouse event, with a ripple after each press (0.4 s of app
  time). `x=&y=` places it without an event; `show=0` hides it. Off by
  default.

A typical run: launch as above, check `status`, `cursor show=1`,
`cap start path=...`, `step frames=30`, then for each action locate the
target with `snap q=path:...`, inject it, and `step` the frames it should
take; `cap stop`; `quit`.

## GPU runs, hidden windows, and the headless backend

- The proof rig for anything visual is the native GPU backend (Metal on
  macOS) in an owned `--remote` instance. `MAKEPAD_HIDE_WINDOWS=1` keeps its
  windows off the user's screen; grabs and input still work because `g`
  forces a present on a hidden window. Measured on macOS (2026-09-10): the
  first hidden `g` answers in ~100 ms, later ones in ~2 s each, and `gseq`
  delivers at that ~2 s spacing regardless of `every_ms`; on an app that is
  not redrawing, `gseq` can time out ("grab timeout"). Rest/settle timing
  proofs need a window that presents on its own (the user's, or a visible
  unfocused one).
- `MAKEPAD_HAL_BACKEND=headless` runs the HAL headless backend: the native
  client with no window, no GPU and no raster. It is for logic and
  data-structure tests only. Never use it to prove a picture or to chase a
  rendering bug.
- Known remote hazards: a hidden-window click is occasionally lost
  (`Event::MouseDown` never arrives) — relaunch before debugging the widget;
  tick-sampled keys need `key k=down` … ≥150 ms … `key k=up`, a `press` lands
  between ticks; in the code map every `g` drops keyboard focus, click the
  map before the next key batch; `MAKEPAD_HIDE_WINDOWS` is implemented on
  macOS and Windows. Test instances of apps with audio or a shared home run with
  `SANDBOX_MUTE=1` and their own `SANDBOX_HOME` / `--state-dir`.

## Coordinate and lifecycle details

### Sharing an instance with a person

Read `activity` before driving an app. `status` includes the same `activity`
object. It reports `user_active`, the monotonic `user_seq`, `idle_ms`, the
two-second `quiet_ms`, whether input is `held`, and the last input's kind and
window. It never exposes typed text, key values or pointer coordinates.
Only native input advances the counter; remote and Studio injections are marked
remote, including hardware-path mouse injection and file drops. Focus, layout
and paint notifications do not count as interaction.

Carry `if_user_seq=N` from the beginning of an automation sequence on every
mutating input command. A command without it is gated by the quiet period alone;
with it, the command is also refused once the person has interacted since N.
Input, window changes, closing/quitting, tweaker mutations, AI overlay
mutations and shader patches are refused with an `{"err":"user_interacting"}` or
`{"err":"user_intervened"}` line (with `"applied":false`) if the person is
active or the counter differs. This check happens on the UI thread immediately
before dispatch. Queued mutations that outlive their request deadline expire.
Read-only status, snapshots, logs and grabs remain available.

User input does not revoke authorization to test or restart an app in the active
development workflow (user instruction, 2026-09-22). If input interrupts a
sequence, discard its interrupted evidence, inspect the reply's `applied` flag
and current state, then read a fresh counter and continue without requesting a
handoff. Wait for the protocol's quiet period when it refuses an input request;
do not replay an already-applied toggle or edit blindly. Graceful close and
replacement launches remain authorized. `gq` also rechecks its counter after
its grabs; if it refuses to quit, inspect the state and retry with a fresh
counter. These permissions apply to the current workflow's app, not unrelated
user instances.

Answers carry no user-sequence headers. Read `activity` before a bounded
sequence and again after it: a changed counter means the person interacted
during it; compare the ending counter with the sequence's original counter
before attributing a capture to your test. A `wait=1` command interrupted before
its frame acknowledgment answers `user_intervened` with `applied:true`; its action already
ran, so retrying it could duplicate an effect. A command refused before dispatch
answers `applied:false`.

- Rectangles are layout points, window-local, with Y increasing downward.
  Window `sz` is logical size; `px` is physical pixels. No DPI arithmetic
  is needed to turn a widget rectangle into a click.
- Window ids are stable slots. A human-closed window reports
  `window N closed by user`; its closure is not a crash. Launch a replacement
  when needed for the active workflow, unless the user has asked to stop.
- Remote input is injected through the app event loop and does not need OS
  focus. Remote windows have a `[remote]` title suffix by default;
  `--remote-title-tag=NAME` customizes it.
- Grabs read the app's own drawable. Files are stored under the startup
  grab directory, with a bounded retained history per window.
- Backend support varies. Inspect the current implementation and response
  rather than assuming every platform supports readback.
- The Studio websocket protocol remains an internal implementation detail;
  agent inspection uses the Unix-socket `makepad-remote` endpoint.

The existing [remote smoke script](../../tools/remote_smoke.sh) exercises
the protocol across example apps. Inspect its current launch/setup behavior
before using it for a task.

For design feedback, styling and structural source edits (`design <op>`),
see [Tweaker](tweaker.md).