makepad/docs/agents/app-remote.md

App remote control

Use this reference when inspecting or driving a Makepad app. Ownership, focus, and capture policy live in AGENTS.md.

The implementation is platform/src/remote.rs. Fetch GET / from your running instance for its current route list; it can contain newer diagnostics than this guide.

Launch and discovery

Build the package in release mode, then launch the resulting standalone binary with --remote. For a hidden verification run from the repo root:

cargo build --release -p makepad-example-splash
MAKEPAD_HIDE_WINDOWS=1 ./target/release/makepad-example-splash --remote

Capture the launch output using your process tool. The startup line contains everything needed to address and clean up that instance:

[makepad-remote] listening on 127.0.0.1:53412 pid=9931 app=makepad-example-splash grabs=/tmp/makepad-remote/makepad-example-splash-9931

The port is ephemeral. --remote=PORT pins it; MAKEPAD_REMOTE=1 or MAKEPAD_REMOTE=PORT also enables the service. Prefer the explicit launch flag for agent inspection. On macOS --focus brings the app to the front as its window opens; a binary launched from a terminal otherwise stays behind the launcher.

Do not discover a port by taking over another running instance. Retain your own launch's PID and endpoint. On code changes, close that instance, rebuild, and relaunch before inspecting again.

Common routes

The core routes use GET. JSON replies are one line; help/tree dumps are text and raw grabs are PNG. Errors carry an err field.

Route	Purpose
/, /help	Current protocol help
/s?w=ID	App/PID and window geometry; omit w to list windows
/g?w=ID&scale=0.5, /gseq?n=8&every_ms=50&scale=0.5	Capture the next presented frame, or N frames on request cadence (1–64, ≥8 ms, ≤60 s span); Metal/raw readbacks downsample before worker PNG encoding; return absolute png path(s) and per-frame capture_ms/encode_ms (gseq.frames); input wait=1 remains a next-frame barrier
/g?raw=1	Return PNG bytes instead of a path
/gq?scale=0.5	Grab windows, then quit; returns png paths and quit:1
/snap?q=TEXT&w=ID&all=1	Widget ids/types/text and window-local rectangles
/d, /dump	Indented widget tree
/m?k=click&x=X&y=Y	Mouse input; kinds also include move, down, up, scroll
/click?x=X&y=Y	Click alias
/k?k=press&c=KeyA	Key input; down/up are also supported
/t?t=TEXT, /k?t=TEXT	Text/IME input
/drop?path=ABSOLUTE_PATH&x=X&y=Y	One file through native drag, drop, and drag-end events
/log?n=50&since=N	App log tail; n in the reply is the latest sequence
/close?w=ID	Close one window normally
/quit	Graceful shutdown without a final grab

/snap entries include window_id and enabled, plus selected when the widget exposes a selection, alongside the compact i/ty/r/w fields.

Query parameters are optional unless needed for the operation. Input routes accept w=ID to target a window and wait=1 to answer after the next frame. With no window specified, routes generally use the first window.

Mouse buttons: b=0 left, b=1 right, b=2 middle. Scroll takes dx and dy. Add hw=1 when testing the hardware pointer-lock/pin transform. Keyboard modifiers are shift=1, ctrl=1, alt=1, and cmd=1.

POST with a flat JSON body is supported. The input parser also accepts long names such as window, kind, button, text, and code. Use curl --get --data-urlencode for arbitrary text in query strings.

File drops require an absolute path (at most 4096 bytes) and finite, nonnegative x/y inside the window. Optional parameters are w/window and wait=1; duplicate or unknown parameters are rejected. The remote thread sends only the path; the app owns validation and loading. The reply reports drop_handled and drag_response, which confirm event handling, not that an asynchronous file import has completed. For example:

curl --get --data-urlencode 'path=/absolute/path/reference car.png' \
  --data-urlencode 'x=300' --data-urlencode 'y=400' --data-urlencode 'wait=1' \
  'http://127.0.0.1:53412/drop'

Drive an owned instance

After replacing the example port with the one your app printed:

curl --fail --silent --show-error 'http://127.0.0.1:53412/'
curl --fail --silent --show-error 'http://127.0.0.1:53412/snap?q=press_demo'

Read the returned r: [x, y, width, height], calculate its center, and feed that point to /click?...&wait=1. Do not reuse coordinates from another window or an earlier layout.

curl --fail --silent --show-error 'http://127.0.0.1:53412/log?n=20'
curl --fail --silent --show-error 'http://127.0.0.1:53412/gq?scale=0.5'

Read a returned PNG with the local image viewer when visual inspection is needed. Confirm the owned process exits after cleanup. Use /quit if a backend cannot grab; do not replace a failed grab with an OS screenshot.

GPU runs, hidden windows, and the simulated-GPU backend

• The proof rig for anything visual is the native GPU backend (Metal on macOS) in an owned --remote instance. MAKEPAD_HIDE_WINDOWS=1 keeps its windows off the user's screen; grabs and input still work because /g forces a present on a hidden window. Measured on macOS (2026-09-10): the first hidden /g answers in ~100 ms, later ones in ~2 s each, and /gseq delivers at that ~2 s spacing regardless of every_ms; on an app that is not redrawing, /gseq can time out ("grab timeout"). Rest/settle timing proofs need a window that presents on its own (the user's, or a visible unfocused one).
• MAKEPAD=gpusim builds the simulated-GPU backend (cfg(gpusim), platform/src/os/gpusim/): a CPU raster that writes frames to files with no window, no Metal shader compile and no presentation. It is for logic and data-structure tests only. Never use it to prove a picture or to chase a rendering bug, and never build it into the shared target/ (CARGO_TARGET_DIR=target-gpusim).
• Known remote hazards: a hidden-window click is occasionally lost (Event::MouseDown never arrives) — relaunch before debugging the widget; tick-sampled keys need /k?k=down … ≥150 ms … /k?k=up, a press lands between ticks; in the code map every /g drops keyboard focus, click the map before the next key batch; MAKEPAD_HIDE_WINDOWS is implemented only on macOS. Test instances of apps with audio or a shared home run with SANDBOX_MUTE=1 and their own SANDBOX_HOME / --state-dir.

Coordinate and lifecycle details

Sharing an instance with a person

Read /activity before driving an app. /s includes the same activity object. It reports user_active, the monotonic user_seq, idle_ms, the two-second quiet_ms, whether input is held, and the last input's kind and window. It never exposes typed text, key values or pointer coordinates. Only native input advances the counter; HTTP and Studio injections are marked remote, including hardware-path mouse injection and file drops. Focus, layout and paint notifications do not count as interaction.

Carry if_user_seq=N from the beginning of an automation sequence on every mutating request. A request without it is gated by the quiet period alone; with it, the request is also refused once the person has interacted since N. Input, window changes, closing/quitting, tweaker mutations, AI overlay mutations and shader patches return HTTP 409 if the person is active or the counter differs. This check happens on the UI thread immediately before dispatch. Queued mutations that outlive their request deadline expire. Read-only status, snapshots, logs and grabs remain available.

User input does not revoke authorization to test or restart an app in the active development workflow (user instruction, 2026-09-22). If input interrupts a sequence, discard its interrupted evidence, inspect the reply's applied flag and current state, then read a fresh counter and continue without requesting a handoff. Wait for the protocol's quiet period when it refuses an input request; do not replay an already-applied toggle or edit blindly. Graceful close and replacement launches remain authorized. /gq also rechecks its counter after its grabs; if it refuses to quit, inspect the state and retry with a fresh counter. These permissions apply to the current workflow's app, not unrelated user instances.

Every HTTP response, including raw PNGs, carries X-Makepad-User-Seq-Start and X-Makepad-User-Seq. A changed counter means the person interacted during the request; compare the ending counter with the sequence's original counter before attributing a capture to your test. A wait=1 command interrupted before its frame acknowledgment returns 409 with applied:true; its action already ran, so retrying it could duplicate an effect. A command refused before dispatch returns applied:false.

• Rectangles are layout points, window-local, with Y increasing downward. Window sz is logical size; px is physical pixels. No DPI arithmetic is needed to turn a widget rectangle into a click.
• Window ids are stable slots. A human-closed window reports window N closed by user; its closure is not a crash. Launch a replacement when needed for the active workflow, unless the user has asked to stop.
• Remote input is injected through the app event loop and does not need OS focus. Remote windows have a [remote] title suffix by default; --remote-title-tag=NAME customizes it.
• Grabs read the app's own drawable. Files are stored under the startup grab directory, with a bounded retained history per window.
• Backend support varies. Inspect the current implementation and response rather than assuming every platform supports readback.
• The Studio websocket protocol remains an internal implementation detail; agent inspection uses the standalone HTTP surface.

The existing remote smoke script exercises the protocol across example apps. Inspect its current launch/setup behavior before using it for a task.

For design feedback and styling, see Tweaker.