9d2e17e3b9
- README: a build quick-start for macOS and Windows, the honest Linux
story, what CUDA is for and how to install the separation model.
- AGENTS.md: the `--remote` control surface protocol, so the harness it
documents is usable without reading platform/src/remote.rs.
- Cargo.toml: workspace membership for the crates this series adds and
removes.
- tools/: the Windows box scripts (wincmd, winps and friends, winrun) and
remote_smoke.sh — how a build gets driven on a remote machine.
- apps/asset-server: the standalone server binary and its README.
- Small follow-ups in libs/{windows,apple_sys,makepad_test,mbtile_reader,
converse} and apps/route, plus .gitignore and makepad.splash.
156 lines
7.2 KiB
Markdown
156 lines
7.2 KiB
Markdown
# `makepad-asset-server` — the standalone Asset Server host
|
|
|
|
The asset store is inherently **multiplayer**. asset-ui, the VJ, the sandbox
|
|
and headless workers are all clients of one catalog, and they open and close
|
|
independently of each other. So the catalog's lifetime must not be any one
|
|
client's lifetime.
|
|
|
|
Historically the Asset UI *embedded* the server: it took `<root>/server.lock`,
|
|
served the catalog, ran the chat broker and the events hub — and every time
|
|
that window was closed or rebuilt, every other connected client lost the
|
|
store mid-session (`503 state unavailable`). This binary is the fix: the same
|
|
service, in its own process, that survives every window.
|
|
|
|
```
|
|
cargo build --release -p makepad-app-asset-server
|
|
./target/release/makepad-asset-server
|
|
```
|
|
|
|
With no flags it serves the checkout's standard store root
|
|
(`local/asset-ui/asset-server`, or `$AI_CONTENT_ASSET_ROOT`) on ephemeral
|
|
ports, announces itself on the LAN beacon, publishes
|
|
`local/ai_content_library`, and coordinates fleet jobs. `--help` lists every
|
|
flag.
|
|
|
|
---
|
|
|
|
## What this process carries
|
|
|
|
| Part | Where it lives | Notes |
|
|
| --- | --- | --- |
|
|
| Catalog + CAS, control/data planes | `AssetServer::start` | search, blobs, ranges/ETags, retire |
|
|
| Auth, tokens, grants | ” | `--root/admin-token` bootstrapped at start |
|
|
| Job queue, worker/lease protocol | ” | plus `GET /v1/job-profiles` |
|
|
| Chat broker (+ client-executed tool parking for game sessions) | ” | `--chat-fleet` / LAN fleet discovery |
|
|
| Games publish path, operations, import routes | ” | |
|
|
| Committed events hub (`/v1/events`) | ” | every client's live view |
|
|
| LAN discovery beacon | ” | `--no-beacon` to stay silent |
|
|
| Lease janitor + bounded blob GC | ” | runs whether or not anyone polls |
|
|
| **ai-content library publisher** | `makepad_asset_importer::watch` | `--library` / `--no-library` |
|
|
| **GPU-fleet job coordinator + profile announce** | `makepad_asset_importer::gen_service` | `--no-jobs` / `--no-announce` / `--fleet` |
|
|
|
|
The last two are the loops the Asset UI used to run *only while it was
|
|
hosting*. They are headless — no `Cx`, no window, no GPU surface — so they
|
|
belong beside the server, and running them here is what makes a UI-less
|
|
deployment a complete fleet citizen. Without the coordinator, jobs any client
|
|
enqueues sit at "waiting for agent" forever.
|
|
|
|
## What deliberately stays client-side
|
|
|
|
These are **content-deriving** loops that need resources only a UI process
|
|
has. They reach the catalog as ordinary clients; moving them here would mean
|
|
giving a headless daemon a window.
|
|
|
|
- **Thumbnail / preview backfill** (`apps/asset-ui`: `thumbnail_renderer.rs`,
|
|
the splat + mesh offscreen renders). Needs a real `Cx`, render passes and
|
|
GPU readback.
|
|
- **Classic-game import** (`import_classic.rs`) and **pack import** wizards —
|
|
interactive, and their icon bakes are GPU renders. The one-shot headless
|
|
equivalents already exist as `makepad-asset-importer --import-pack /
|
|
--import-games / --import-music / --import-ai-library` when a batch run is
|
|
wanted.
|
|
- **Stems / lyrics analysis bake** (`analysis.rs`) — a large local model on
|
|
the operator's machine, driven from the surface that asks for it.
|
|
- **Generation pipelines** driven from the Create surface (`pipeline.rs`) —
|
|
these are *requests*; the queue and the dispatch live server-side.
|
|
|
|
## Recommended deployment
|
|
|
|
One daemon, every app attached:
|
|
|
|
```bash
|
|
# 1. the store (leave it running; it survives every UI restart)
|
|
./target/release/makepad-asset-server > /tmp/asset-server.log 2>&1 &
|
|
|
|
# 2. every client, told never to hold the root itself
|
|
export ASSET_UI_ASSET_EMBED=never
|
|
./target/release/makepad-app-asset-ui
|
|
./target/release/makepad-vj
|
|
```
|
|
|
|
`ASSET_UI_ASSET_EMBED=never` (aliases: `no`, `off`, `0`, `false`, `attach`,
|
|
`client`) makes the Asset UI a pure client: it never takes
|
|
`<root>/server.lock`, at startup or during a succession. Without it the app
|
|
still attaches when the daemon holds the root — but whichever process starts
|
|
*first* wins the lock, so a UI launched before the daemon would put the store
|
|
back inside a window.
|
|
|
|
The VJ and the sandbox are already pure clients; they need no flag.
|
|
|
|
### How a client finds the server
|
|
|
|
Three ways, in order of precedence:
|
|
|
|
1. `ASSET_UI_ASSET_SERVER=ip:controlport:dataport` — an explicit pin.
|
|
2. `<root>/listen`, `<root>/server-id`, `<root>/admin-token` — what the
|
|
daemon writes at startup. This is the attach path, and it is why the
|
|
default bind is `0.0.0.0:0`: the *file* is the address of record.
|
|
3. The LAN discovery beacon — how a client on another machine finds it with
|
|
no configuration at all.
|
|
|
|
## What survives what
|
|
|
|
- **Closing / rebuilding / crashing any app** — the server keeps serving.
|
|
Every other client stays connected; nothing is lost.
|
|
- **Restarting the daemon** — attached clients notice the silence within
|
|
~3.5 s, say so honestly in the status chip, and rejoin the new process as
|
|
soon as it rewrites `listen`. Cursors and leases are re-established; the
|
|
events journal reports an explicit `gap` rather than a silent hole.
|
|
- **A daemon crash with `EmbedPolicy::Auto`** — an attached Asset UI succeeds
|
|
it (takes the lock, starts the loops, reconnects to itself). This is the
|
|
transition-mode safety net, not the deployment.
|
|
- **A daemon crash with `ASSET_UI_ASSET_EMBED=never`** — no app fills the
|
|
vacancy; clients wait and rejoin whoever takes the root next. Restart the
|
|
daemon.
|
|
- **Two daemons on one root** — refused immediately and by name
|
|
(`server root: locked by another server process`). One process per root is
|
|
a law: the job routing metadata assumes a single enqueuer, and two writers
|
|
over one WAL catalog would be two sources of recovery truth.
|
|
|
|
## Bouncing the daemon safely
|
|
|
|
```bash
|
|
# find it (it holds the lock and wrote the listen file)
|
|
cat local/asset-ui/asset-server/listen # ip:control:data
|
|
pkill -TERM -f 'makepad-asset-server' # clean shutdown: joins every thread
|
|
./target/release/makepad-asset-server > /tmp/asset-server.log 2>&1 &
|
|
```
|
|
|
|
SIGTERM/SIGINT shut down in order: background loops first (so nothing is
|
|
still publishing into a closing catalog), then the planes, then the state
|
|
thread. Startup logs what recovery found — `recovered N cas temps / M leases`
|
|
— which is the honest report of what the previous life left behind.
|
|
|
|
## Isolated instances
|
|
|
|
Never point a scratch instance at the live root, and never let one beacon:
|
|
|
|
```bash
|
|
./target/release/makepad-asset-server \
|
|
--root /tmp/scratch-store --work /tmp/scratch-work \
|
|
--no-beacon --no-jobs --no-library
|
|
```
|
|
|
|
`--no-beacon` keeps peers hunting for the real store from finding the scratch
|
|
one; `--no-jobs` keeps it from claiming the real fleet's queued work.
|
|
|
|
## Tests
|
|
|
|
`cargo test -p makepad-app-asset-server --release` covers the composition:
|
|
the host serves a catalog over real sockets and shuts down cleanly, a second
|
|
host on one root is refused by name, a missing library directory never costs
|
|
the catalog its server, and the defaults are the deployment defaults. The
|
|
parts themselves are covered where they live — `libs/asset/store` (HTTP,
|
|
chat, events, jobs, security, operations suites) and `libs/asset/importer`
|
|
(watch, coordinator, gen-service). The attach/succession contract is in
|
|
`apps/asset-ui/src/asset_store_state.rs`.
|