andodeki/makepad
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
makepad/docs/agents/app-remote.md
Admin c27ee1dccb tools, ci, docs, workspace: the Builder ships as source on Windows and is one shell script on macOS and Linux, the CI box, makepad-agents, cargo-makepad, and one workspace for every crate 
The Makepad Builder (tools/makepad_builder: the Windows download is makepad-builder.zip with the Builder's source; macOS and Linux run makepad.sh), the CI runner (tools/ci), cargo-makepad, the agents tool, the root workspace and the documents.

Covers work 6f1e44649..2b1a41df4 outside platform, draw, widgets, libs, apps and examples:

- wm: the Android super-app compiles tiles on the phone against the host engine
- cargo-makepad: the super-app APK is packed by `android dyn-pack`, in Rust
- widgets, storybook: the widget catalogue app, some eighty new widgets, a theme store with mixable style sheets, and the rule that a press belongs to
- wm, widgets: the style tween starts with all eight weights, and the new style sheet takes the last number instead of macOS's
- platform: the Android desk swipes at 120 Hz — offscreen render passes and framebuffers cached, SVG meshes kept, font families that complete
- cef: a page's audio can be captured instead of played
- Button: activate from the keyboard when focused (#1241)
- libs/ai: exclude hub_ui and services from the AI workspace (#1242)
- flow: the flow-ui app and the flowgraph canvas return to the workspace, with staged progress in libs/flow
- svg: a stroke that bends tighter than its half width collapses its inner edge to the corner
- the accent a name is written with stops deciding whether search finds it
- macOS: keep the paint clock beating while the window is minimized (#1244)
- Linux: only link libdrm for the direct backend (#1245)
- cargo-makepad: find dependency crate dirs via cargo metadata (#1246)
- app_main!: only ship the fonts an app declares (#1247)
- ai stems: LaneDemixer, VocalsModel and the platform Stage origin/main needs
- platform, widgets, wm: every Vulkan pass renders through a negative-height viewport again, so a Wayland or X11 window stands upright, and no capture
- calendar: its own colour contrast is named explicitly, now that the theme store exports one under the same name through the widgets glob
- widgets: the supersampled resolve and the map's shadow mask sample their textures as stored, like every other render texture
- platform: every OpenGL texture pass stores top-left rows, custom cameras included, and culls with the winding its inverted projection gives
- platform, wm, director: a hosted app's software frame arrives as a top-left texture and the window manager and the director carry no flip term at al
- widgets, platform: the last twelve warnings go
- The Builder compiles itself again from the Makepad tree Scope's release pins and removes the source snapshots nothing is built from any more
- The regex engine accepts \b and \B, reports the earliest match so a scan can resume after it, and can decide word boundaries on ASCII so a code sear
- workspace: Scope's settings carry the code map's tab width and the directories each prepared project hides from its map
- AGENTS.md: a platform change made for one application needs the user's feedback and checks first, and app specifics never leak into the shared layer
- wm: the desktop style menu lists each style once
- cef, video, platform: what Stage's browser needs from the shared layers
- macOS: re-arm the display links when a window leaves the Dock (#1251)
- platform, wm: work builds again on iOS, tvOS and wasm, and the window manager is warning-free on Windows
- ci: tools/ci, a Splash runtime that watches a branch and runs every ci.splash it finds
- stitch: the crate says what the rest of the repository says about its license
- ci: a run is one target dir and one cargo batch per target, and the wall draws
- ci: the wall shows THIS run and is fit for an OLED
- ci: untested is grey and nothing else
- ci: warming fills the cache and blames nobody
- Vulkan: stop printing the loader's startup narration by default (#1252)
- ci: the terminal gets its pty helper, director is a desktop tool, and a refused input is asked again
- platform: no warnings on tvOS and linux_direct, and a missing pty helper says so
- apps: what the CI box found
- ScrollBar: add `show_handle` for a view that scrolls without a grabbable bar (#1254)
- cargo-makepad: keep the android SDK at a stable path (#1253)
- ci: the window is a dashboard
- Remove Flow, asset and VJ applications moved into Stage
- ci: a stop or a restart is not a test result
- ci: a quiet header, a build that visibly moves, and scripts that leave the machine alone
- widgets: children a lookup discovers reach the dump, the snapshot and the flood searches
- tests: the workspace suite, run as a whole for the first time, passes outside the example UI tests
- platform: a hidden window wears no hands-off frame
- examples: the UI tests pass, for the reasons they failed
- ci: a library's warnings turn the workspace block yellow, and a desktop tool is not warmed for the web
- libs: no warnings in the workspace check on any row
- ci: the workspace script gives the wall back after warming
- ci: a judgement is never lost to the way it was phrased, and every "retry" of the bridge is retried
- tests: the last three failures of the CI box's night
- Linux: fix window chrome button hovers and how maximized/fullscreen windows work (#1255)
- ci: the header names the tip being tested
- tools: makepad-screen is makepad-agents, and its binary is `agents`
- gif: the crate's doc examples compile
- ci: the tests are the platform's own, in release, in two minutes
- Tooltip: position anchored tooltips in the same draw (#1256)
- cef: a hosted browser survives its host's exit
- ci: a card is as high as its content
- tests: the three binaries over ten seconds in release come under it
- widgets: let the host install script mods into every Splash isolate
- widgets: a pooled test context forgets the last case's Escape claim
- widgets: the pooled test context's resets replace the globals
- wm: the shell menu says what it did, and the CI waits for that
- platform: the storage module says how much a volume has free
- git: the memory ledger can take what it is asked for
- ci: a driven app takes every key once, and the storage module builds for the browser
- RadioButton: fix its touch hover state and click-off behavior (#1258)
- HtmlLink: fix its hover and pressed states (#1257)
- RadioButton: take key focus on the click, not on the press (#1259)
- script: re-entrant dispatch, thread index validation, Any-based handle downcasts, UTF-8 previews; regex: never_loop fix
- script: port Octoscript WS1 — worklist equality with fuel/deadline/work bail, uncaught-error bail, allow_debug_output
- script: port Octoscript parser/tokenizer/control-flow VM patches (ws2)
- script: port Octoscript heap/string/array/object hardening onto upstream's allocation budget
- script: port orphaned Octoscript VM hardening (execution caps, clear_type_methods, parser diagnostics)
- script: keep silenced streaming evals running past uncaught errors
- script: the Octoscript hardening keeps a Makepad host's semantics and its speed
- fix(text): enable ttf-parser gvar-alloc for many-tuple variable fonts
- feat(text): CoreText outline fallback for hvgl-only fonts (macOS)
- draw text: the CoreText outline fallback is only ever resolved for a face whose outlines live solely in hvgl, and font-family diagnostics are the `f
- mail: the phone layout lines up
- clock: the phone layout fits both orientations
- calendar, reminders, calculator, sheets: the phone layouts fit their space
- platform, widgets, wm: a touch that is taken away is cancelled, never released
- weather, notes, finance, photos, route: the phone layouts tidy up
- files: the phone layout tidies up
- wm: the phone shell's surfaces match the app and the grid
- wm on Android: every app is its own process
- clock: a cancelled release neither opens an alarm nor switches tab
- platform, wm: hosted apps ask the WM for what they cannot do themselves
- wm on Android: apps build on the phone
- wm on Android: app transitions and gestures feel like the phone's
- android: rotated Vulkan windows stop rebuilding their swapchain every frame, and hosted children get touch
- vulkan (android): a window released on suspend is not released again
- android: hosted children hand frames over with GPU fences and draw only when they have work
- wm on Android: gestures and surfaces settle like the phone's
- widgets: touch lists scroll with Android's physics
- wm on Android: the shell follows the Pixel launcher's motion
- wm on Android: Recents shows the apps over a receding home, and the home screen switches looks
- wm on Android: the look switcher stays put, the iOS dock shows its icons, background apps follow a look change
- task, cargo-makepad, widgets, audio: the task manager chooses its columns, graphs every process' network and disk traffic and installs itself as a D
- agents: a user touching an app under test no longer stops the agent
- flowgraph: ordered ports take many wires, sockets can override their icon, and the canvas embeds as a viewport
- cef: captured frames and audio carry callback clocks and a navigation epoch
- git: timings read a portable clock
- ai: the hub's rig-fixture tests find the asset library without the asset client
- ai llm: a Metal main buffer is filled in 32 MB chunks
- diskmap: a scan lists each macOS directory in one getattrlistbulk call and classifies files without allocating or locking
- files: the tile view's three projections are toolbar buttons
- mail: the local view takes a theme-derived palette and line icons, search pages share records
- director: agent lanes form a tree and Grok joins the usage bar
- widgets, draw: skeuomorphic surfaces light each other through an optional relief buffer
- widgets, platform: a texture can light the relief buffer, menus take colour chips, and a style reload recompiles changed shader functions
- widgets: relief surfaces can travel like mechanical keys, and dark surfaces take less neighbour light
- platform, widgets, audio_route, ai: window crossfades and whole-frame presents, caption controls that click, themed menus, a stereo-pair audio tap,
- builder: the Builder TUI matches the new terminal design, Windows executables carry app icons, and every app has one
- platform: Android builds without Vulkan compile again, and the font-selection test counts the hosted entry point
- platform: the pipeline-skip repaint mark exists only on Apple, where Metal reads it
- builder: the terminal scripts delete nothing and read top to bottom, and every Builder delete stays inside its own folder
- files: the Files app deletes nothing and never overwrites
- director: coding agents ask before acting unless you choose otherwise
- sheets, score, git, home: saves cannot cut a file short or overwrite another one, and checkouts cannot write outside the repository
- git: worktree status matches git on real repositories and can be cancelled
- builder: one forward-only bar per setup component, green checks on finished setup rows, a black Builder window and a smaller Windows ZIP
- builder: the GPU notice is remembered
- cuda: builds link only the CUDA toolkit they are given, never a system install
- builder: a refused email opens the editor again with what was typed and says why, and the app section is YOUR APPS
- builder: a refused email says "Email not recognised" and keeps what was typed for fixing
- platform (windows): every window keeps animating with several windows open
- builder previews: long work gets a page of its own (steps as a checklist, the current one carrying its bar, footer working · ctrl+c stops), Account
- builder: long work runs on its own page
- builder: a first build resolves online, the Rust row and bars move when the work does, and the window fits the TUI
- audio_route: one API on macOS, Windows and Linux, monitoring by default and processing only when asked, plus shaders that discard a value compile to
- platform, widgets, ai: a restyle recompiles nothing it already has, shows only complete frames, and a window can own its caption
- builder: Compile shows a real bar
- builder: the compile line is the bar and n / total crates
- builder: apps get the icon their package declares
- builder: no Scope command
- widgets: relief surfaces can swing their light toward a point
- builder: a HEAD request is a HEAD
- builder: the menu waits for a choice
- builder: no menu flash after log-in
- builder: wait out Windows security on fresh Rust
- builder: the menu opens on the first of YOUR APPS
- platform (windows): half-float RGBA textures have four channels
- builder: CUDA whenever the machine has NVIDIA
- audio_route example: print the samples that reached the processor each second
- widgets, builder: the big widget families are features, all on by default, and the Builder takes none of them
- platform (macos): the display link is kept until it is invalidated
- ai hub: a local chat with no model says so once, plainly
- builder: the Windows exe is built for size
- windows: desktop apps open no console window, and still speak through pipes
- ai: Claude Desktop drives any app with the F10 panel
- Constrain Splash external I/O to the host service bridge (#1243)
- PortalList: stop following the end when scrolling to an earlier item (#1261)
- wm: starts on Windows
- ci: a full disk empties the build output before the run, not every row after it
- builder: an incremental compile's bar follows what really compiles
- builder: the compile bar's total is the app's own crates
- builder: menus wrap around
- wm: children open no console windows on Windows
- wm: the whole deck out of a Builder's source, compiled only when the person opens an app
- builder: CUDA crates build in an installation whose path has a space
- ai: Connect hands the .mcpb to Claude Desktop itself
- wm: the clock and calendar know the date on Windows
- platform (windows): a popup opened for the first time draws its rows
- wm: opening an app rebuilds it when its binary is out of date; warm instances never build
- builder: an app edited after its build shows as needing a compile
- civil-time: one local wall clock for every app, and it knows the zone on Windows
- platform + wm: a hosted child redraws at its tile's new size
- platform (windows): a texture pass drawn before its window has a size is skipped, not a crash
- draw: glass of a view that stopped drawing leaves the screen
- calendar: the Calendars sheet is an opaque panel, and the wide layout does not float one over its sidebar
- build: dev keeps line tables only, release is incremental without LTO, the parallel frontend is a documented local opt-in
- cargo-makepad: a binary with several app_main! entry points bundles
- script: #[derive(Script)] emits one helper call per field, and ScriptNew's default methods keep their bodies out of every type
- widgets: the Widget lookup methods and with_script_vm_id are compiled once, not once per widget type
- platform: studio-protocol no longer waits for script, so platform starts ~0.5 s earlier
- ai-speech: makepad-ai-sfx (and with it ai-h3) is built only for IndexTTS
- platform: an app's package dir, icons and bundle name rebuild only that app
- script + wasm_bridge: no build script that reruns on every file
- build: a private app cloned into apps/<name> joins this workspace, Stage first
- build: the AI stack, csg, Scope and Source Library join the one workspace
- builder: every app builds in the one Makepad workspace, one target directory per source snapshot
- ai chat: the chat's backends are the CLI, MCP and cloud providers; the local model is the `localai` feature
- apps: every app that hosts the chat has a `localai` feature for the local model, off unless the app's own job runs one
- builder: time each install component and phase
- builder: Build tools, Windows SDK, Rust and CUDA install side by side, each with its own bar
- builder bootstrap: rustc, rust-std and cargo download and unpack side by side under one Rust bar
- widgets: the library moves to makepad-widgets-core in widgets/core, and makepad-widgets becomes its front crate
- widgets: the Window, widget tree and panel theme reach the tweaker, voice, AI slot and dock through hooks, not through their modules
- widgets: every widget family is a crate of its own under widgets/families, and makepad-widgets links, re-exports and registers the ones its features
- aichat, widgets: the chat reads the design feedback through the tweaker's hook, so linking the chat no longer needs the tweaker
- apps, examples, libs: each crate builds only the widget families it uses, and the design overlay is each app's own default feature
- builder: an app release builds without the app's development defaults, and the catalog names every feature a release ships with
- widgets: the build scripts rerun on their own edits only, and makepad-widgets no longer reads MAKEPAD
- livepipe: takes makepad-widgets without its default families, now that the AI stack is in the one workspace
- builder: the catalog names the new localai defaults of route and ai-hub
- feedback: Send feedback, a caption icon and a small panel that shows exactly what is sent
- builder: an app it launches knows who the person is, for Send feedback
- feedback: the panel's wording reads right and draws in any font
- feedback: takes makepad-widgets without its default families
- builder: clear build data deletes target/ itself and says why when it cannot
- ai chat: a build without local AI starts on the first provider that answers here, not on "No model"
- platform: MAKEPAD=gpusim builds again
- cargo-makepad: a Windows desktop build links the app's icon into its own binary, not into RUSTFLAGS
- thiserror: RUSTC_BOOTSTRAP no longer turns on the unstable generic member access API
- builder: every end-user build uses rustc's parallel frontend, and falls back to one thread when it fails
- builder: the Windows SDK downloads the 25 cabinets its MSIs name instead of all 149, no screen or log shows a \\?\ path, long status messages wrap,
- builder: CUDA kernels compile in any install folder, and when they cannot the app is built without them
- builder: makepad-builder.exe declares itself like a well-formed Windows program and stops a stalled build through a job object instead of taskkill
- platform (macos): a contained panic no longer frees the windows under AppKit or leaves an app that ignores quit
- builder: macOS and Linux run the Builder as one shell script, the same TUI as Windows, and nothing is compiled for it
- builder + ai-cuda: the CUDA kernel progress shows in the Builder's Compile row, not scribbled over its screen
- builder: an app built with CUDA starts on Windows
- builder (windows): the downloaded exe is built large again
- builder (unix): the email check takes real addresses
- repo: no third-party comic archive, generated models or sample asset in the tree
- remove big example media files
- cargo_makepad: no stray KNMI radar frame in the Android Java sources
- widgets: the empty parts of an app's caption bar drag the window again
- ScrollBar: fade out when idle, like macOS overlay scrollers (#1264)
- platform: every app builds the same `windows` crate
- wm: in a Makepad Builder installation apps build through the Builder, however wm was started
- platform, draw: web builds start again
- platform: web text draws again
- widgets: the keyboard reaches boxes and modals
- widgets: a hover tooltip never sits under the mouse cursor
- platform: clipboard_read, reading text or an image from the system clipboard
- feedback: Send clipboard, and the dialog works from the keyboard
- builder: the Windows Builder ships as source, runs in its own console, and apps compile with Rust's GNU toolchain or Microsoft's
- platform: an app also finds its resource map in the builder folder beside it
- win_resource: PNG decode and encode on makepad-fast-inflate instead of zune-png
- builder: one folder layout everywhere, makepad-builder.zip with only the files a Windows build reads, a Rust download over parallel connections, and
- builder (windows): the .bat runs from a folder with spaces and parentheses, and the ZIP always carries it with CRLF
- builder: coding agents start from the macOS/Linux Builder, its compile bar has its end, the app rows say where each app is in a word or two, and Win
- builder (unix): running curl | sh again opens the existing installation with its saved email
- builder: log out from the Account row
- builder: `makepad-builder build APP` compiles an app offline exactly as the Builder does

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 20:11:31 +02:00

195 lines
10 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# App remote control
Use this reference when inspecting or driving a Makepad app. Ownership,
focus, and capture policy live in [AGENTS.md](../../AGENTS.md).
The implementation is [platform/src/remote.rs](../../platform/src/remote.rs).
Fetch `GET /` from your running instance for its current route list; it can
contain newer diagnostics than this guide.
## Launch and discovery
Build the package in release mode, then launch the resulting standalone
binary with `--remote`. For a hidden verification run from the repo root:
```sh
cargo build --release -p makepad-example-splash
MAKEPAD_HIDE_WINDOWS=1 ./target/release/makepad-example-splash --remote
```
Capture the launch output using your process tool. The startup line contains
everything needed to address and clean up that instance:
```text
[makepad-remote] listening on 127.0.0.1:53412 pid=9931 app=makepad-example-splash grabs=/tmp/makepad-remote/makepad-example-splash-9931
```
The port is ephemeral. `--remote=PORT` pins it; `MAKEPAD_REMOTE=1` or
`MAKEPAD_REMOTE=PORT` also enables the service. Prefer the explicit launch
flag for agent inspection. On macOS `--focus` brings the app to the front as
its window opens; a binary launched from a terminal otherwise stays behind
the launcher.
Do not discover a port by taking over another running instance. Retain your
own launch's PID and endpoint. On code changes, close that instance, rebuild,
and relaunch before inspecting again.
## Common routes
The core routes use GET. JSON replies are one line; help/tree dumps are text
and raw grabs are PNG. Errors carry an `err` field.
| Route | Purpose |
|---|---|
| `/`, `/help` | Current protocol help |
| `/s?w=ID` | App/PID and window geometry; omit `w` to list windows |
| `/g?w=ID&scale=0.5`, `/gseq?n=8&every_ms=50&scale=0.5` | Capture the next presented frame, or N frames on request cadence (1–64, ≥8 ms, ≤60 s span); Metal/raw readbacks downsample before worker PNG encoding; return absolute `png` path(s) and per-frame `capture_ms`/`encode_ms` (`gseq.frames`); input `wait=1` remains a next-frame barrier |
| `/g?raw=1` | Return PNG bytes instead of a path |
| `/gq?scale=0.5` | Grab windows, then quit; returns `png` paths and `quit:1` |
| `/snap?q=TEXT&w=ID&all=1` | Widget ids/types/text and window-local rectangles |
| `/d`, `/dump` | Indented widget tree |
| `/m?k=click&x=X&y=Y` | Mouse input; kinds also include move, down, up, scroll |
| `/click?x=X&y=Y` | Click alias |
| `/k?k=press&c=KeyA` | Key input; down/up are also supported |
| `/t?t=TEXT`, `/k?t=TEXT` | Text/IME input |
| `/drop?path=ABSOLUTE_PATH&x=X&y=Y` | One file through native drag, drop, and drag-end events |
| `/log?n=50&since=N` | App log tail; `n` in the reply is the latest sequence |
| `/close?w=ID` | Close one window normally |
| `/quit` | Graceful shutdown without a final grab |
`/snap` entries include `window_id` and `enabled`, plus `selected` when the
widget exposes a selection, alongside the compact `i`/`ty`/`r`/`w` fields.
Query parameters are optional unless needed for the operation. Input routes
accept `w=ID` to target a window and `wait=1` to answer after the next frame.
With no window specified, routes generally use the first window.
Mouse buttons: `b=0` left, `b=1` right, `b=2` middle. Scroll takes `dx`
and `dy`. Add `hw=1` when testing the hardware pointer-lock/pin transform.
Keyboard modifiers are `shift=1`, `ctrl=1`, `alt=1`, and `cmd=1`.
POST with a flat JSON body is supported. The input parser also accepts long
names such as `window`, `kind`, `button`, `text`, and `code`. Use
`curl --get --data-urlencode` for arbitrary text in query strings.
File drops require an absolute path (at most 4096 bytes) and finite,
nonnegative `x`/`y` inside the window. Optional parameters are `w`/`window`
and `wait=1`; duplicate or unknown parameters are rejected. The remote
thread sends only the path; the app owns validation and loading. The reply
reports `drop_handled` and `drag_response`, which confirm event handling,
not that an asynchronous file import has completed. For example:
```sh
curl --get --data-urlencode 'path=/absolute/path/reference car.png' \
--data-urlencode 'x=300' --data-urlencode 'y=400' --data-urlencode 'wait=1' \
'http://127.0.0.1:53412/drop'
```
## Drive an owned instance
After replacing the example port with the one your app printed:
```sh
curl --fail --silent --show-error 'http://127.0.0.1:53412/'
curl --fail --silent --show-error 'http://127.0.0.1:53412/snap?q=press_demo'
```
Read the returned `r: [x, y, width, height]`, calculate its center, and feed
that point to `/click?...&wait=1`. Do not reuse coordinates from another
window or an earlier layout.
```sh
curl --fail --silent --show-error 'http://127.0.0.1:53412/log?n=20'
curl --fail --silent --show-error 'http://127.0.0.1:53412/gq?scale=0.5'
```
Read a returned PNG with the local image viewer when visual inspection is
needed. Confirm the owned process exits after cleanup. Use `/quit` if a
backend cannot grab; do not replace a failed grab with an OS screenshot.
## GPU runs, hidden windows, and the simulated-GPU backend
- The proof rig for anything visual is the native GPU backend (Metal on
macOS) in an owned `--remote` instance. `MAKEPAD_HIDE_WINDOWS=1` keeps its
windows off the user's screen; grabs and input still work because `/g`
forces a present on a hidden window. Measured on macOS (2026-09-10): the
first hidden `/g` answers in ~100 ms, later ones in ~2 s each, and `/gseq`
delivers at that ~2 s spacing regardless of `every_ms`; on an app that is
not redrawing, `/gseq` can time out ("grab timeout"). Rest/settle timing
proofs need a window that presents on its own (the user's, or a visible
unfocused one).
- `MAKEPAD=gpusim` builds the simulated-GPU backend (`cfg(gpusim)`,
`platform/src/os/gpusim/`): a CPU raster that writes frames to files
with no window, no Metal shader compile and no presentation. It is for
logic and data-structure tests only. Never use it to prove a picture or to
chase a rendering bug, and never build it into the shared `target/`
(`CARGO_TARGET_DIR=target-gpusim`).
- Known remote hazards: a hidden-window click is occasionally lost
(`Event::MouseDown` never arrives) — relaunch before debugging the widget;
tick-sampled keys need `/k?k=down` … ≥150 ms … `/k?k=up`, a `press` lands
between ticks; in the code map every `/g` drops keyboard focus, click the
map before the next key batch; `MAKEPAD_HIDE_WINDOWS` is implemented only
on macOS. Test instances of apps with audio or a shared home run with
`SANDBOX_MUTE=1` and their own `SANDBOX_HOME` / `--state-dir`.
## Coordinate and lifecycle details
### Sharing an instance with a person
Read `/activity` before driving an app. `/s` includes the same `activity`
object. It reports `user_active`, the monotonic `user_seq`, `idle_ms`, the
two-second `quiet_ms`, whether input is `held`, and the last input's kind and
window. It never exposes typed text, key values or pointer coordinates.
Only native input advances the counter; HTTP and Studio injections are marked
remote, including hardware-path mouse injection and file drops. Focus, layout
and paint notifications do not count as interaction.
Carry `if_user_seq=N` from the beginning of an automation sequence on every
mutating request. A request without it is gated by the quiet period alone;
with it, the request is also refused once the person has interacted since N.
Input, window changes, closing/quitting, tweaker mutations, AI overlay
mutations and shader patches return **HTTP 409** if the person is active or
the counter differs. This check happens on the UI thread immediately
before dispatch. Queued mutations that outlive their request deadline expire.
Read-only status, snapshots, logs and grabs remain available.
User input does not revoke authorization to test or restart an app in the active
development workflow (user instruction, 2026-09-22). If input interrupts a
sequence, discard its interrupted evidence, inspect the reply's `applied` flag
and current state, then read a fresh counter and continue without requesting a
handoff. Wait for the protocol's quiet period when it refuses an input request;
do not replay an already-applied toggle or edit blindly. Graceful close and
replacement launches remain authorized. `/gq` also rechecks its counter after
its grabs; if it refuses to quit, inspect the state and retry with a fresh
counter. These permissions apply to the current workflow's app, not unrelated
user instances.
Every HTTP response, including raw PNGs, carries `X-Makepad-User-Seq-Start`
and `X-Makepad-User-Seq`. A changed counter means the person interacted during
the request; compare the ending counter with the sequence's original counter
before attributing a capture to your test. A `wait=1` command interrupted before
its frame acknowledgment returns 409 with `applied:true`; its action already
ran, so retrying it could duplicate an effect. A command refused before dispatch
returns `applied:false`.
- Rectangles are layout points, window-local, with Y increasing downward.
Window `sz` is logical size; `px` is physical pixels. No DPI arithmetic
is needed to turn a widget rectangle into a click.
- Window ids are stable slots. A human-closed window reports
`window N closed by user`; its closure is not a crash. Launch a replacement
when needed for the active workflow, unless the user has asked to stop.
- Remote input is injected through the app event loop and does not need OS
focus. Remote windows have a `[remote]` title suffix by default;
`--remote-title-tag=NAME` customizes it.
- Grabs read the app's own drawable. Files are stored under the startup
grab directory, with a bounded retained history per window.
- Backend support varies. Inspect the current implementation and response
rather than assuming every platform supports readback.
- The Studio websocket protocol remains an internal implementation detail;
agent inspection uses the standalone HTTP surface.
The existing [remote smoke script](../../tools/remote_smoke.sh) exercises
the protocol across example apps. Inspect its current launch/setup behavior
before using it for a task.
For design feedback and styling, see [Tweaker](tweaker.md).
Reference in a new issue View git blame Copy permalink