LocalModels runs registry models in-process (install state, resumable downloads, recorded licence acknowledgements at $MAKEPAD_HOME/license_acks.json, weight paths by file role) and libs/ai/hub_ui is the install panel + licence modal every app can embed. New native ports: Beat This! (beats + downbeats) and Basic Pitch (note transcription) with their registry entries; the Salamander Drumkit samples (CC BY-SA 3.0, 37 files pinned by size and sha256) as a sample bank the downloader fetches like a model.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The last two hub commits staged whole files and carried uncommitted hunks
of another lane (beats-native, notes-native, the local runner, new
domains and license keys) that reference files not yet in the tree. This
restores those files to the body changes only; the other lane's edits
stay in its working tree.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The native port covers everything the Python reference worker did (body,
hands, mask prompt, multi-person), so the subprocess backend, its fake
worker harness, the sam3dbody-ref registry entry and the
MAKEPAD_SAM3DBODY_* environment go. The packet validator moves to the
native backend.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The sam3dbody backend reads its options from the request's prompt string
(`hands`, `detect`, `persons=N`): `hands` runs the full mode and the
packet carries which hands were fused and their boxes; `detect` finds up
to N persons with SAM 3.1 (an optional native-segment role on the body
entry, the same artifact the segment entry pins) and runs one body pass
per person with its box and mask, so the packet's people array grows.
The body crate shares one body pass between the packet, mask and hands
entry points, and infer_full takes the mask prompt too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BodyModel::infer_masked takes a provided person box and its full-frame 0/1
mask: the mask is warped with the crop's own affine (bilinear, rounded back
to 0/1 like the reference's uint8 warp), encoded by the prompt encoder's
mask CNN (four stride-2 convolutions with channel LayerNorm and erf GELU,
then a 1x1 to 1280) and added to the backbone tokens before the ray
conditioning, replacing the folded no-mask term. Oracle parity on the
mask fixture: warp exact, every CNN stage within f32 accumulation noise,
conditioned context 8e-4, end to end kp3d 2.2 mm / kp2d 0.25 px. 41 tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BodyModel::infer_full runs the reference's full mode natively: the body
pass's hand boxes become two 512 crops (the left one cut from the mirrored
image, padding 0.9), each goes through the backbone, its own ray
conditioning and the hand decoder (the body decoder with the *_hand
tensors), the hand-mode rig pre-transform (local_to_world_wrist, wrist and
root offsets, non-hand parameters zeroed) and the hand camera head's
scale factor of 10; the left result is un-mirrored. The fusion gates each
hand (wrist angle, box size, keypoint spread, wrist distance), re-prompts
the body decoder with the trusted wrists and elbows as point prompts (the
decoder now takes N prompt tokens and a previous estimate), and writes the
fused wrist angles, hand parameters and hand scale/shape back before the
final rig pass. pose.rs gains the roma xyz (extrinsic) and XZY (intrinsic)
euler pairs; mhr.rs returns joint global rotations.
Oracle parity on both full-mode fixtures: every hand-decoder stage per
step (tokens 7e-3, heads 4e-4, rig params 1e-4), fusion reports as the
reference (one hand trusted on the standing photo, both on the crop), and
end to end kp3d 1.6 mm / kp2d 0.85 px. 36 tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One backend call per SAM-style decoder layer (PE norms, token
self-attention, token-to-image cross-attention, erf-GELU feed-forward,
final norm) inside one command buffer, with the layer's f32 weights cached
on the device under their content identity and pooled transients.
gpu_two_way_layer_resident in the common backend; CUDA declines and the
per-op path stays. Body decoder loop on the M3 Max: layers 100 -> 9 ms,
frame 405 -> 266 ms, oracle parity unchanged (kp3d 1.5 mm).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gpu_linear_f32_resident promised a resident weight but the Metal tensor
backend re-uploaded it every call, which for the MHR rig's 664 MB pose
corrective matrix cost 50 ms a step. Metal tensors now carry a content
identity (fresh at creation and after every in-place write, so a cache
keyed by it can never serve a stale weight), and the resident linear caches
the device buffer under it. Body frame on the M3 Max: 705 -> 405 ms (rig
302 -> 20 ms, refine 41 -> 13 ms, heads 10 -> 6 ms).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Metal tensor backend is host-Vec based: every op copied its inputs up
and its result back, which put the DINOv3 backbone at 2.1 s a frame on an
M3 Max. try_vit_backbone_resident_f32 runs a whole pre-norm ViT stack
(LayerNorm, q/k/v, rotate-half rope from tables, flash attention, out,
LayerNorm, SwiGLU, down; residuals in place) inside one command buffer
against cached bf16 weights, with a new kernel_makepad_rope_half_tables_f32
kernel for the rope. The precompiled metallib now carries the bf16 GEMM
kernels the runtime source compile already enabled on bfloat devices.
The common backend exposes it as gpu_vit_backbone_resident (CUDA declines;
its per-op path is already resident) and the body backbone tries it first:
233 ms a frame on the M3 Max, same oracle parity (kp3d 1.5 mm).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The FP8 backbone mode added each linear's bias by downloading, tiling and
re-uploading it per call, which cost more than the FP8 GEMM saved. A
gpu_add_cols_broadcast op (CUDA kernel; host loop on the Metal tensor
backend) adds a cols-wide bias to every row on the device, and each linear
keeps its bias resident after the first upload.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BodyModel::set_backbone_fp8 quantises every backbone weight to E4M3 with
a per-tensor absmax scale and runs the tensor-core FP8 GEMM (bias
broadcast after). A backend without FP8 turns it off per layer on the
first refusal, so Metal keeps bf16. The oracle test reports its
accuracy and timing next to bf16; the default stays bf16 until the
numbers say otherwise.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The backbone's linears now take the cuBLASLt bias-epilogue / bf16 mm
paths (bf16 operands, f32 accumulation, bf16 output — the reference's
precision) with the f32-accumulating GEMM as the fallback. The rig's
pose correctives run on the final refinement step only by default: the
intermediate steps only feed keypoints back into the decoder, and the
oracle shows the difference is 0.4 mm (2.1 vs 1.7 mm against the
reference) for half the loop's rig time; `correctives_every_step` keeps
the exact mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The backbone, conditioning and decoder now take any square crop whose
side is a multiple of 16 (512 is the trained size); BodyModel::set_crop_size
selects it and caches the dense positional grid per size. Measured on the
oracle image against the reference: 512 gives 1.7 mm mean keypoint error,
384 and 256 about 2 cm mean (6 to 7 cm worst joint), 192 falls apart —
the knob is a real accuracy trade, not free speed. The decoder loop
reports its split (layer chain, heads, rig+camera, refinement) so the
next optimisation is chosen on numbers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Apps ask the hub for a recognizer or a voice and get one; where it runs is
the hub's decision. AiHub::start_stt / start_tts return poll-driven
sessions shaped like the chat session. The Auto ladder is Whisper/Kokoro in
this process (weights present, machine election), on the machine node over
loopback, on a LAN node, else the OS engine; SpeechReach::Local is the
"don't reach out" knob. Audio always comes back as PCM: the app owns the
device.
Three layers:
- makepad-ai-speech is the whole speech model family, engines only.
libs/voice (Whisper + Silero VAD) folds in as the `whisper` and `vad`
modules next to kokoro and indextts, each a cargo feature; the Apple
bridges and the Speaker/VoiceTranscriber selection leave it.
- makepad-system-speech (new) is the OS speech services as blocking fns:
Apple SpeechAnalyzer/AVSpeechSynthesizer via Swift, Windows.Media.Speech*
on the vendored bindings, Android SpeechRecognizer/TextToSpeech through
MakepadSpeech.java (API 26 floor), espeak-ng on Linux. It models the two
STT shapes honestly: PCM in (Whisper, Apple) versus an engine that owns
the microphone (Android, Windows), with capabilities the caller reads.
- the hub grows speech sessions, in-process Whisper/Kokoro workers with the
residency election, a `whisper` wire backend (stt domain, registry entry
pinned to ggerganov/whisper.cpp) so a Mac can serve a Quest, and a
`language` field on the generate request.
Consumers: the Window voice input runs on an STT session and switches to
engine-mic mode when the recognizer owns the microphone; converse's
SpeechOutput is a lazily started TTS session plus a pump thread; route
drops its private speech copy for converse; vj's lyrics fallback and the
alignment bakes call the engines directly.
Verified here: speech-roundtrip through the real sessions (Apple voice in,
in-process Whisper on Metal out, 4.3% WER); system-speech-test TTS->STT
verbatim; hub/converse/system-speech unit tests; msvc, aarch64-android and
linux-gnu cross-checks; Java against android-34. Windows, Android and Linux
bridges are compile-checked only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The backbone (20 heads) and the decoder (8 heads) both have 64-wide
heads, which the stack's FA2 kernel covers: f16 operands with f32
softmax and accumulation, the reference's own precision class, instead
of the composite path that materialises the 1029 x 1029 scores per head.
The composite path stays as the fallback where a backend lacks the
kernel. Oracle parity unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The ray-conditioning conv used to download the 1024 x 1280 embedding,
concatenate the 99 ray features on the host and upload the result; it is
now two resident linears over the two column blocks with the no-mask
term folded into the bias, plus an add (32 ms -> a few on the 4090).
The decoder loop downloaded the whole normalised token block every
layer for its one pose row; it now slices that row and fetches the
block once at the end (or per layer under a trace). Same numbers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On the 4090 the frame was 885 ms of which 840 were host scalar loops in
the decoder loop: the three 70-row refinement FFNs (about 240M
multiply-adds a step), the two heads, and the rig's 55k x 3000
corrective output layer plus its identity blendshape sum. Those are now
GPU-resident linears (heads.rs GpuStepHeads, MhrRig::prepare_gpu), the
skinning computes one transform per joint instead of one per influence,
and the results are identical (all 29 oracle tests unchanged). The rig
still runs entirely on the host when no GPU side was prepared.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The recut in a6341981d applied its patch against the wrong directory
and restored the six shared files to the previous tree without the
body-native hunks. This commit adds exactly those: the `body-native`
feature and optional dependency, the pinned `sam3dbody` registry entry
and its test, the backend arms and the module declaration. Working
tree untouched.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
66e5e2f11 committed the working tree of the shared hub files and with it
another lane's uncommitted edits (a new domain, request fields, a
backend arm, manifest lines). This commit restores those files to the
previous tree plus only the body-native hunks. The working tree is
untouched: the peer's edits stay on disk as their uncommitted work,
exactly as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
model.rs closes the loop: crop -> backbone -> ray-conditioned context ->
six decoder steps, each through the pose head, the rig, the camera and
the projection, then the packet the sandbox already reads (kp3d/kp2d in
camera axes, the 204 rig parameters, global rotation, camera translation,
joint positions). Against the reference on the oracle image, on Metal:
3D keypoints within 1.7 mm, 2D within 0.4 px, rig parameters, camera
and rotation within 2e-3. packet.rs writes the JSON by hand with the
reference worker's rounding and field order.
The hub gains the `body-native` feature (default on): registry entry
`sam3dbody` pinned to the Comfy-Org repack by revision, size and sha,
body_native_backend.rs beside the subprocess reference backend with the
same live_step contract, the `body` capability advertised when the
feature is compiled, and a stubbed test double for the CPU-only tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
decoder.rs builds the 145-row token set (pose, previous, prompt, two
hand-box rows, 70 keypoint rows, 70 3D-keypoint rows), runs the six
layers with the SAM-style repeated positional encoding (none on the
first layer's self-attention), cross-attends against the ray-conditioned
context, and after each step hands the normalised pose token to the
heads and updates the keypoint rows from the caller's feedback: the
2D-keypoint positional FFN and the bilinearly sampled context features
on the valid rows, the pelvis-centred 3D positional FFN on the rest.
heads.rs holds the host-side ReLU FFN heads, the refinement FFNs, the
hand-box MLP and the hand classifier. Against the oracle on Metal, every
layer's residual stream is within 6e-3, every step's pose head within
8e-4 and camera head within 2e-5; token assembly and the host heads
match to 1e-4.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dino.rs is the 32-block ViT-H+/16 on the gpu_* surface after the TRELLIS
conditioner: bf16-resident linears with f32 accumulation, layer scale
folded into the output projections, rotate-half rope, SwiGLU, the
model's own final norm. preprocess.rs is the crop (box -> 1.25 pad ->
3:4 -> square -> 512 bilinear warp, ImageNet normalisation), the CLIFF
condition vector and the patch rays; condition.rs is the dense
positional encoding and the ray-conditioned decoder context. Against the
reference oracle on Metal: backbone 1.1% mean relative (bf16 noise),
ray-conditioned context 8e-5, crop within one u8 rounding step, rays and
dense PE 1e-7. One reference detail the paper does not state: its shrink
of the ray field is an antialiased filter whose taps clip at the image
edge, so the two edge patches sample inside their block centre (9.03 and
501.97 rather than 7.5 and 503.5); block centres left 0.1 of error.
The Metal tensor backend was missing rope_half (it aliased the
interleaved layout) and layer_norm_mul_add; both now exist with the
CUDA contract, which is what lets this backbone run on Apple silicon.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mhr.rs is the Momentum Human Rig on the CPU: blendshapes, the 889x249
parameter transform, parents-first similarity kinematics (x,y,z,w
quaternions, Rz Ry Rx eulers, 2^s scales), the sparse-then-dense pose
corrective MLP over joints 2.., linear blend skinning and the 308-row
keypoint regression. pose.rs decodes the 519-wide head output: 6d
rotations, the 23 ball / 58 hinge / 6 translation body layout, the
mixed-dof hand layout, scale and hand component bases, the camera
translation and the perspective projection. Against the reference
oracle: vertices within 1e-4 cm with correctives on, keypoints within
1e-6 m, rig parameters within 1e-7. Two conventions the spec could not
settle on paper are now settled by the fixture: the head's global
rotation triple arrives Z,Y,X-ordered, and the corrective features start
at joint 2 (750 wide). Fixture tests skip cleanly without the oracle
directory or the weights.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
makepad-ai-body joins the AI model workspace: the constants of the
architecture (DINOv3 ViT-H+/16 at 512, the 6-layer promptable decoder,
the 519-wide pose head, the 127-joint / 18439-vertex MHR rig) and the
single-file safetensors reader for the Comfy-Org repack, which fails
closed on a Meta checkpoint-style header and checks the shapes the port
is written for at load. The backbone, decoder and rig modules follow
in their own lanes against the spec under local/agent_state/sam3dbody.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A realtime feed session parked in its wait-for-a-frame loop forever when
the client died without sending stop (a sandbox quit left job-2 live on
.123 for five minutes holding the GPU slot). The wait loop now returns
to the top of the session loop once no socket is left, where the idle
timeout counts a socketless session down. Test covers it.
SkinnedModel gains node_parent/node_count, joint_skinned_centroid (the
direction a leaf limb actually runs, from the flesh it skins) and
from_nodes (a mesh-less rig for hierarchy-maths tests) — what the
sandbox's webcam mocap retarget needs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B626urtY1Xo4hQdLzvSK6F
New `body` backend (sam3dbody-ref): a persistent length-prefixed-PNG /
JSON-lines worker subprocess seam with ready handshake, per-frame timeout,
bounded restarts. LiveFrameOut grows aux_json — structured per-frame JSON
sent to the client before the frame — and output_encoding "none" makes a
session pose-only (refused with loop_mode feedback, also on control flips,
which upgraded apply_control to Result). Worker code+model stay
box-provisioned via MAKEPAD_SAM3DBODY_WORKER; the repo carries only the
MIT seam. Codex lane + Fable review (ready handshake, spawn timeout).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0165w1ZL1f1TruX5u2qC7mSX
the vision and ocr backends' ContentBackend machinery lives behind the
llm feature; the imports and the lane-refill helpers now do too, so a
client crate compiling the hub with a leaner feature set (vj, chat-ui)
builds without warnings. pipeline_submit_collect stays feature-free for
its tests and says so.
Squashed from work:
- asset-ai: FastH3 4-step fast video backend; clip keyframes on the wire
- asset-ui: loop video chains — text→image→video that ends where it began
- h3: safetensors -> pruned-Q4_K GGUF quantizer for the 24GB DiT tiers
- h3_quant_gguf verify: row-error gates calibrated to the measured Q4_K floor
- asset-ai realtime: the feedback loop — the source anchors, the drifted frame inits
- asset-ai realtime: a feedback loop survives a resize and travels by default
- asset-ai realtime: the feedback loop frees itself from the feed handshake and pauses for its listener
- asset-ai realtime: the outbound encode leaves the loop's critical path
- asset-ai ocr: the ocr domain — Chandra 2 at page resolution, and the tower goes planner-owned
- llm slots: a lane can hold an image span — embedding prefill and a rope cursor of its own
- vision tower on CUDA: the encode leg gets its two missing kernels
- llm/ocr: one M-RoPE grid encoder for both image paths, and a livelock made an error
- vision tower on CUDA: the f16 GEMM keeps the precision it was throwing away
- live: a feed that moves box takes its trip with it — one seed image
- vision tower on CUDA: the tiled attention becomes bit-exact, and tensor cores go
- llm prefill on CUDA: the MMA attention kernel gets the tile a 4-to-1 model needs
- asset-ai ocr: the CUDA encode lane joins the integration — vision-parity sits beside run's three arms, and the kernels
- Merge branch 'ocr-perf-integration' into work
- asset-ai: the live anchor can follow the trip, and text leaves the 5090
- asset-ai: the camera moves the world, and the world starts still
- asset-import: the EA strategy classics, in the one 2D contract
- rtsmap: one seeded generator for tiled strategy maps
- asset-ui: one card for the strategy classics, with a pack dropdown
- asset-ai: music3 reference-audio path, ocr/h3 backends, registry
- asset: mp4 sample index for range-streaming, chat tools, import profiles
- cnc: tiberium is twelve growth frames, not twelve empty variants
- platform: native file and save dialogs, in-house on all three desktops
- chat: the scan holds out for a lane home
- chat: a full home queues you — take the free lane
- chat: the preload has a percentage, and the boundless cap stops showing
- llm cuda: the 32x2 attention tile — even GQA ratios stay on MMA
- sa3 gets a bake path: the sfx model's tables precomputed by a diffusion-side bin
- sqlite_query: anti-join regression test
- td import: HARV's second frame block is its harvesting cycle, not a turret
- asset-ui: sprite enhancement runs on the 32B dev DiT — distillation, not the prompt, was the ceiling
- ai-hub: makepad-asset-ai becomes makepad-ai-hub at libs/ai/hub, the chat pane becomes makepad-chat-ui, the service bin
- asset-ui: test health fixtures grow the realtime field they were born without
- ai-hub: one home at ~/.makepad — weights/ run/ cache/ logs/, the service cache migrates from ai_content by a single re
- ai-hub: subprocess workers die with the node — process groups everywhere, PDEATHSIG on linux, one KILL_ON_JOB_CLOSE Jo
- ai-hub: the hub object — AiHub::in_process, pipes vocabulary, and the local LLM engine generalized out of mpfiles (aic
- strict-json: the dependency-free JSON module gets its own crate; asset-client re-exports it so nothing downstream move
- ai-hub: the machine layer — node entries, the 0600 machine token, and the residency election that IS the lock (aicore
- ai-hub: MPHUB1 — the fabric beacon only dedicated nodes can send (aicore §4)
- ai-hub: job leases — work lives only while it is renewed (aicore §8)
- asset-creator: the pipeline library is born — specs, the deps gate, and the derived-state law (aicore §9)
- ai-hub: RAM residency facts — the CPU-side twin of residency.rs (aicore §3)
- ai-hub: ETA placement primitives — relative GPU throughput, the four-term estimate, and an observable breakdown (aicor
- ai-hub: leases go live on the wire — origin fields on submit, /job/<id>/keepalive, /bye, and the reaper that cancels w
- ai-hub: the chat providers move in — fleet qwen, openai, grok, claude/codex/grok CLIs, the responses driver, and the w
- asset-creator: the engine — one pipeline run against the hub, deps-gated, spliced, cancellable, resumable-by-construct
- ai-hub: the machine node mode — --machine binds loopback, registers in ~/.makepad/run, and exits on its own once idle
- asset-creator: makepad-creator-run — the detached client for runs that must outlive a window (aicore §9)
- ai-hub: a native Claude Messages-API provider — API-key or Claude Code OAuth, bounded SSE streaming, injected tools (a
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- asset-creator: the preset tables move in — fifteen chain-policy constants shared by every creator app (aicore §9 / P6)
- makepad_ai is deleted — every backend is a hub pipe, the agent seam lives with its consumers (aicore §14, decided 2026
- ai-hub: loads hold the machine residency election — set_model_state claims on Loaded and publishes the service port (a
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- ai-hub: pick_for_domain_eta — ETA-ranked placement over the shared hard-filter core (aicore §6 / P4)
- asset-creator: the engine picks a provider per stage at dispatch time — a chain's later stages see fresh fleet state (
- ai-hub: the fabric secret gates the service HTTP surface — bearer on everything but /health and the ticketed peer path
- vj: DREAM runs execute in the app — pipelines.rs becomes the run it used to watch (aicore §9 / F1)
- asset-creator: the runner — generate one thing and put it in the catalog, one implementation for every surface (aicore
- chat-ui: the session runs in the app — no broker anywhere on the chat path (aicore P8 / F5)
- asset-store: assets.query is a first-class query endpoint — the bounded SQL surface outlives the broker (aicore P8 / F
- asset-creator: CreatorTools — the chat tool pack for a store that only stores (aicore §9 / P8)
- asset-store: the shrink — the store stores (aicore P7)
- importer + asset-server host: the coordination era ends (aicore P7)
- store config purge + asset-ui goes fleet-direct; the derive protocol gets its route proof (aicore P7)
- client + chat dispatcher: the dead wire comes out (aicore P7/P8)
- ai-hub: 0.3.0 — the health version says which era a node runs
- ai-hub: the default fleet is 'gen' — apps hear the LAN without env plumbing
- ai-hub: the preload note percents the prefill, not the job bar
- ai-hub: conversations keep their KV — the wire mirror, the lane identity, the in-turn dynamic context (aicore §7)
- ai-hub: an open-think model is thinking from its first token
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
- zero-warning sweep, round two — the first full-workspace pass
- zero-warning sweep, round three — the model lanes and the deep examples
- zero-warning sweep, round four — the last stragglers
- zero-warning sweep, round five — vj and chat-ui
- zero-warning sweep, round six — three cascades
Co-authored-by: Claude <info@makepad.nl>
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- work: land the sources the last commits reference
- kenney: catalogue all 50 free 3D kits; Modal dismissed() never fired
- asset store: central vision-annotation queue; Kenney donate prompt
- asset-ai: vision domain — image + prompt -> text on every fleet node
- annotation runs the fleet's vision services through the normal job queue
- vj: responsive DJ mixer + Windows drag-and-drop, cherry-picked from PR #1199 (vjroger)
- store search: WordNet synonym expansion, query-side; per-term seeks
- video scheduling: a cold model pin never downloads past a warm one, and a dedicated box only serves its role
- an expansion can never lose a run — and `expand: true` finally means something
- fleet panel: a slow box is not a missing box
- h3: first+last keyframe conditioning (the weights were always FL2VA)
- registry: the three H3 FL2VA tiers name their real conditioning
- fleet scheduling: spread before stacking, and a stuck job moves
- a download never steals a job from a box that has the weights
- the faster GPU takes the tie: 6000 > 5090 > 4090
- an evicted flux model gives the card its VRAM back
- a job now records what each of its stages was handed
- open a stage in RUNS and read what it sent
- every expanded song was an instrumental: the lyrics had nowhere to go
- the writer never overwrites words the person wrote
- store: a dependent job's body is spliced from its deps' results at claim
- a fleet box's job row says what that job was asked for
- the store runs a whole pipeline, and one record says how far it got
- a client can declare a whole run, watch it, and stop it
- 100% now means published, and the expander is a job you can queue
- one card says what a spawned task is doing, everywhere
- a run that ends says so, instead of being noticed later
- when every box holding the model is busy, buy another copy
Squashed from work; the fine-grained history is under tag archive/work-2026-08-26:
- vj: the tweener learns — RIFE runs on the Mac and feeds the warp
- asset-ai: the chat tells the truth while it works
- llm: a cold turn on the solo slot takes the session-native speculative path — think-mode turns no longer re-ingest the whole conversation through the draft head (66 → 122 tok/s on the four-lane box)
- llm: the step cost model is chosen per device — the RTX PRO 6000's measured verify curve (13.7 + 3.17·B ms) beside the 5090's; the bench warms every tail shape and times two windows
- vj: local store, lyrics and model plumbing, and the frame-interpolator's device parity check
A Turing box (RTX 2080 Ti, sm_75) lost ALL of CUDA because two kernel
files refused to compile for it, and one failed kernel build means the
stub store — surfaced in the VJ as "stems: model error: no compiled-graph
device" on the DJ tab.
diffusion_ops.cu used three sm_80-only pieces unguarded: bf16 wmma
fragments (the type itself is incomplete before Ampere), cp.async, and
the m16n8k16 mma shapes. The cp.async helpers now fall back to
synchronous copies below sm_80 — the f16 wmma flash/sdpa kernels lose
their prefetch overlap on Turing, not their contents — while the bf16
and FA2 kernels are compiled out and their launchers refuse pre-sm_80
devices with cudaErrorNotSupported instead of returning a buffer the
kernel never wrote.
fattn/common.cuh made mkllm_unused_vars constexpr: the no-cp.async
branch of ggml_cuda_fattn_mma_get_nstages calls it, and a non-constexpr
callee poisoned the constexpr config chain on exactly the pre-Ampere
device pass — the arch nobody had compiled for.
Stems verified on the 2080 Ti: stems-ops-check all green (SNR 137-147 dB
against the CPU reference), two tracks separated end to end, output
confirmed clean by ear.
Co-authored-by: vjroger <r.deleeuw@qogni.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The model code was spread across eight crates that had grown into each other:
ggml and cuda and mlx each owned part of a tensor runtime, llama and tts and
voice2 each owned part of a model, and libs/diffusion owned everything else.
They are now one tree with an explicit shape:
libs/ai/cuda — kernels and launch surface
libs/ai/metal — Metal shaders and the shim
libs/ai/llm — the language-model runtime (sessions, lanes, contexts,
the CUDA and Metal executors, the compiled Metal path)
libs/ai/models/ — common, flux, h3, music, paint, speech, stems, vision
libs/diffusion is not deleted but demoted: what remains is the VALIDATOR
crate — several dozen `*_validate.rs` oracles that check a native
implementation against a reference, which is where they belong now that the
implementations live next door.
The functional work inside the move is mostly in the LLM runtime: N lanes that
draft while one verify batch serves all of them, per-slot prefill over a shared
folded attention arena, speculation that survives batching, and a scheduler
that reports rather than publishes. And in the CUDA build: a machine without
usable CUDA must still LINK (and say so), the default kernel arch is the
building machine's GPU, `NO_CUDA` forces the stub even where the toolkit
exists, and kernels compile in parallel with progress.
libs/video_flow is new here: classical optical flow estimation and the `mkfl`
motion-field payload — a flow field measured from a clip without a model,
which is what drives free-rate bounce-looping playback and the uprez/tween
enhance pipe.
