libs/sqlite_query reads WITHOUT ROWID tables through the index cursor,
refuses to write them, and the pager releases its process write slot on
drop. libs/tar is an in-repo tar reader with gzip through fast_inflate.
libs/git imports packed objects with bounded checkout writers and its
HTTP file responses carry a trailer. libs/code_language gains a Haskell lexer with literate (Bird) dialect
detection. libs/loader_bundle and libs/search are new;
libs/app_module carries the super-app module surface; libs/workspace
adds Settings.infinite_zoom and RendererChoice::gpu_env_value; libs/ai
builds without warnings across the hub, llm, metal and model crates;
windows-rs job object handles are c_void.
Squashed from work (the libs parts of each):
- Restore retained renderer support for Scope (libs/search)
- Share Builder target across Makepad app builds (libs/loader_bundle)
- Index local Apple Mail with Gmail labels, attachments and reimport (libs/sqlite_query)
- libs/ai: warning cleanup across the hub, llm, metal and model crates
- code_language: a Haskell lexer with literate (Bird) dialect detection
- git: packed imports and bounded checkout writers; HTTP file responses carry a trailer
- builder: dedicated-folder installer checks, catalog and runtime fixes; Windows job objects hold c_void handles (libs/windows)
- workspace: Settings.infinite_zoom, the experimental prepared map inside the glyph
- wm: the Android super-app hosts apps as on-demand dylibs (libs/app_module)
- libs/tar: an in-repo tar reader; the super-app unpacks its archives with it
- workspace: RendererChoice::gpu_env_value follows the platform's runtime GPU choice
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Squash of 54 work commits (Sep 1–12):
6251f7c ai-hub: body domain — live pose packets ride the realtime session
ea50c77 chat_ui: the feed's session gets its profile brief back
f51b5f3 ai-body: the crate for the native SAM 3D Body port, with its weights reader
8211ae6 ai-body: the MHR rig and the pose head's parameter decoding, oracle-exact
9e343a8 ai-body: the DINOv3 ViT-H+/16 backbone, crop and ray conditioning; Metal gains rope-half and affine layer norm
69d842c ai-body: the promptable pose decoder and its refinement loop, oracle-matched on Metal
66e5e2f ai-hub: SAM 3D Body runs natively — `sam3dbody` on the body domain, oracle-matched end to end
a634198 ai-hub: the body-native commit carried a peer's in-flight hub hunks; put them back where they were
9ff44e8 ai-hub: the body-native wiring, this time only the lane's hunks
6a1c16b ai-body: third-party notices — what the port is implemented after, and what it is not
d78411a ai-body: the per-step work moves to the GPU
b22259b ai-body: the context stays on the GPU; only the pose token leaves the loop
346f31f ai-body: flash attention for the head-dim-64 blocks
45b5b98 ai-body: the crop size is a runtime knob, and the loop reports where its time goes
4be6d19 ai-body: the test modules import the grid constants they still use
7598346 ai-body: tensor-core GEMMs for the backbone, and the rig's correctives only where they count
a9ce596 ai-body: the crop warp runs across cores
8964ba6 ai-body: an FP8 backbone mode, off by default, measured against the oracle
a2aaa8f ai-body: the FP8 bias rides a column-broadcast add on the device
d53c77d metal: a device-resident ViT stack, and the body backbone rides it
d006d0a metal: resident f32 linears keep their weight on the device
525ba1c metal: a device-resident two-way decoder layer, and the body decoder rides it
c9e6d88 ai-body: the hands pass — hand crops, the hand decoder, the hand-mode rig and the wrist fusion
62dff26 ai-body: the mask prompt — a person's segmentation mask conditions the body pass
a648cf8 ai-hub: body session options — hands, detect, persons=N
8c568df ai-hub: drop the SAM 3D Body reference worker backend
7ff875a ai-hub: keep a peer's in-flight beats/notes/local work out of the body commits
31e5faa ai-hub: local model runner, licence acknowledgements, a shared install panel; Beat This!, Basic Pitch and the Salamander drum-kit entries
b94bc58 ai-services: the wire, the app port and the panel state — one conversation, many apps
2acb798 ai-services: wire v2 — endpoints, receiver-side caps, result disposition
8ae0ffb ai-services: the engine core — registry, router and conversation, tested against a scripted model
2308736 ai-services: the real models behind the engine feature — local through the hub, Claude, and none
c3f631d livepipe: one reusable pipe from a camera to a fleet node and back
ff62db3 ai libs: the runtime env-var cleanup — precision is a per-caller policy, not an environment side channel
04a94ef realtime: one service-log line when a live session opens and one when it closes
0ecb81c ai models: the model-crates env-var cleanup — 172 research knobs gone, the unset default is the code
4ca36c1 ai hub + services: the assistant's model comes from wherever it is resident — the fleet chat box, with tools, then the local weights
432121e aichat engine + wm: launch, then use — the assistant continues in the same turn once the app it started is on the bus
7a5bf69 ai-hub registry: the Salamander drumkit samples come from the makepad.nl mirror — the GitHub repo only carries the .sfz files
102ffc5 ai-services: messages on the bus — a manifest declares topics, the engine subscribes on a tool's behalf or by ToolResult.subscribe, a service publishes Message frames, an idle conversation wakes on a message as an event turn under rate laws; the WM bus forwards the new frames; every app that matches the wire gets its arm
a837792 hub + flow: a whitespace-only chat completion is retried once and then fails instead of passing as an answer; a flow's model is a fleet model id unless it names a weight file on disk; chat models show under the text domain in /v1/models
bc6c620 hub + flow: what the chat review found — the in-process route retries an empty completion too, a node says whether its prefill opened thinking so a brief-mode answer is never discarded, a preferred model falls back to normal election when no node has it, discovery keeps looking for the preferred model until patience runs out
75c3441 hub: the PRO 6000 serves image as well as chat and text
ad5e98b hub registry: flux2-dev's VRAM estimate is its measured peak, 30 GB
c7241e0 hub: a node that evicted every resident releases its cached allocator pool before refusing a load or publishing usable VRAM
30575f0 flow: route generation by request workload
1be1e21 ai-hub: gate downloads by disk capacity and recover fleet admission
df6b394 filesystem_watcher, bounded_http, ai services: live and tool prerequisites
79ebdb9 ai-hub: add a native Pixal3D image-to-3D backend
0ba0d74 ai-hub: propagate typed refusals under reject queue policy
cc6c872 Speed up H3 conditioning and video decoding
e512059 Fix Qwen vision residency and generated material colors
2864f68 ai-hub http client: bound every plain TCP connect to 3 s per address
3d93229 ai: CUDA is a Linux/Windows-only dependency; the hub library defaults to llm + stt
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 5090 kept about 1.5 GB of CUDA allocator pool after evicting every
model, so a card that fits flux2-dev refused it (30510 MB free reported,
32090 MB in a fresh process). Admission now trims the pool when the last
resident is gone and re-measures before refusing; usable VRAM is measured
after the same trim; the refusal names the pool it released.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The AI runtime crates read their precision, activation and kernel
choices from environment variables; the code path taken with none set
is the one that ships, so every knob that selected it is now an explicit
argument and every losing path is gone. GemmPrecision { f16_accumulate,
f16_activations } is passed by each caller: the default {true, true} is
the old unset Flux route; H3's DiT and text encoder pass {false, false}
(H3's >1e4 activations saturate f16 — the policy H3 used to set through
FLUX_GEMM_F16ACC=0 on itself), its VAE {true, false}; DA3's StrictF32
selects f32 packed attention in code; Hy-Motion carries an explicit
f16_attention_operands flag through its text refiner, its double and
single blocks and the CUDA backend (true in production, false only in
its full validator). The libs/diffusion bins — a separate workspace —
are migrated to the same shapes.
Benches and validators no longer set variables on themselves: llama's
skip-logits is a session option (the CUDA bench turns it on), OCR takes
explicit use_f16_gemm and tiled_roformer options, the lane speculative
probe reads its CLI. The live gates the first cut had deleted are back
as explicit-config tests: MMQ M=129, the strided-f32 MMV path, the
RMS+MUL CPU oracle. The loader's THREADS and CHUNK_MB stay real settings.
The Metal quantized-matmul experiment (metal_qmm and its vendored MLX
kernels) was reachable only through a knob and goes with it.
Reviewed in three rounds by the delegate reviewer (the last round
accepted everything but one Hy-Motion call site, fixed in round four
and reviewed here), and gated on the Windows CUDA box: lib checks of
common/paint/loader/cuda/llm/motion/vision, motion 24 and vision 23
tests, the hub check, the diffusion bins, llm 253 passed / 1 ignored.
On this Mac: the same checks plus the motion and vision tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The FP8 backbone mode added each linear's bias by downloading, tiling and
re-uploading it per call, which cost more than the FP8 GEMM saved. A
gpu_add_cols_broadcast op (CUDA kernel; host loop on the Metal tensor
backend) adds a cols-wide bias to every row on the device, and each linear
keeps its bias resident after the first upload.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Apps ask the hub for a recognizer or a voice and get one; where it runs is
the hub's decision. AiHub::start_stt / start_tts return poll-driven
sessions shaped like the chat session. The Auto ladder is Whisper/Kokoro in
this process (weights present, machine election), on the machine node over
loopback, on a LAN node, else the OS engine; SpeechReach::Local is the
"don't reach out" knob. Audio always comes back as PCM: the app owns the
device.
Three layers:
- makepad-ai-speech is the whole speech model family, engines only.
libs/voice (Whisper + Silero VAD) folds in as the `whisper` and `vad`
modules next to kokoro and indextts, each a cargo feature; the Apple
bridges and the Speaker/VoiceTranscriber selection leave it.
- makepad-system-speech (new) is the OS speech services as blocking fns:
Apple SpeechAnalyzer/AVSpeechSynthesizer via Swift, Windows.Media.Speech*
on the vendored bindings, Android SpeechRecognizer/TextToSpeech through
MakepadSpeech.java (API 26 floor), espeak-ng on Linux. It models the two
STT shapes honestly: PCM in (Whisper, Apple) versus an engine that owns
the microphone (Android, Windows), with capabilities the caller reads.
- the hub grows speech sessions, in-process Whisper/Kokoro workers with the
residency election, a `whisper` wire backend (stt domain, registry entry
pinned to ggerganov/whisper.cpp) so a Mac can serve a Quest, and a
`language` field on the generate request.
Consumers: the Window voice input runs on an STT session and switches to
engine-mic mode when the recognizer owns the microphone; converse's
SpeechOutput is a lazily started TTS session plus a pump thread; route
drops its private speech copy for converse; vj's lyrics fallback and the
alignment bakes call the engines directly.
Verified here: speech-roundtrip through the real sessions (Apple voice in,
in-process Whisper on Metal out, 4.3% WER); system-speech-test TTS->STT
verbatim; hub/converse/system-speech unit tests; msvc, aarch64-android and
linux-gnu cross-checks; Java against android-34. Windows, Android and Linux
bridges are compile-checked only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Squashed from work:
- asset-ai: FastH3 4-step fast video backend; clip keyframes on the wire
- asset-ui: loop video chains — text→image→video that ends where it began
- h3: safetensors -> pruned-Q4_K GGUF quantizer for the 24GB DiT tiers
- h3_quant_gguf verify: row-error gates calibrated to the measured Q4_K floor
- asset-ai realtime: the feedback loop — the source anchors, the drifted frame inits
- asset-ai realtime: a feedback loop survives a resize and travels by default
- asset-ai realtime: the feedback loop frees itself from the feed handshake and pauses for its listener
- asset-ai realtime: the outbound encode leaves the loop's critical path
- asset-ai ocr: the ocr domain — Chandra 2 at page resolution, and the tower goes planner-owned
- llm slots: a lane can hold an image span — embedding prefill and a rope cursor of its own
- vision tower on CUDA: the encode leg gets its two missing kernels
- llm/ocr: one M-RoPE grid encoder for both image paths, and a livelock made an error
- vision tower on CUDA: the f16 GEMM keeps the precision it was throwing away
- live: a feed that moves box takes its trip with it — one seed image
- vision tower on CUDA: the tiled attention becomes bit-exact, and tensor cores go
- llm prefill on CUDA: the MMA attention kernel gets the tile a 4-to-1 model needs
- asset-ai ocr: the CUDA encode lane joins the integration — vision-parity sits beside run's three arms, and the kernels
- Merge branch 'ocr-perf-integration' into work
- asset-ai: the live anchor can follow the trip, and text leaves the 5090
- asset-ai: the camera moves the world, and the world starts still
- asset-import: the EA strategy classics, in the one 2D contract
- rtsmap: one seeded generator for tiled strategy maps
- asset-ui: one card for the strategy classics, with a pack dropdown
- asset-ai: music3 reference-audio path, ocr/h3 backends, registry
- asset: mp4 sample index for range-streaming, chat tools, import profiles
- cnc: tiberium is twelve growth frames, not twelve empty variants
- platform: native file and save dialogs, in-house on all three desktops
- chat: the scan holds out for a lane home
- chat: a full home queues you — take the free lane
- chat: the preload has a percentage, and the boundless cap stops showing
- llm cuda: the 32x2 attention tile — even GQA ratios stay on MMA
- sa3 gets a bake path: the sfx model's tables precomputed by a diffusion-side bin
- sqlite_query: anti-join regression test
- td import: HARV's second frame block is its harvesting cycle, not a turret
- asset-ui: sprite enhancement runs on the 32B dev DiT — distillation, not the prompt, was the ceiling
- ai-hub: makepad-asset-ai becomes makepad-ai-hub at libs/ai/hub, the chat pane becomes makepad-chat-ui, the service bin
- asset-ui: test health fixtures grow the realtime field they were born without
- ai-hub: one home at ~/.makepad — weights/ run/ cache/ logs/, the service cache migrates from ai_content by a single re
- ai-hub: subprocess workers die with the node — process groups everywhere, PDEATHSIG on linux, one KILL_ON_JOB_CLOSE Jo
- ai-hub: the hub object — AiHub::in_process, pipes vocabulary, and the local LLM engine generalized out of mpfiles (aic
- strict-json: the dependency-free JSON module gets its own crate; asset-client re-exports it so nothing downstream move
- ai-hub: the machine layer — node entries, the 0600 machine token, and the residency election that IS the lock (aicore
- ai-hub: MPHUB1 — the fabric beacon only dedicated nodes can send (aicore §4)
- ai-hub: job leases — work lives only while it is renewed (aicore §8)
- asset-creator: the pipeline library is born — specs, the deps gate, and the derived-state law (aicore §9)
- ai-hub: RAM residency facts — the CPU-side twin of residency.rs (aicore §3)
- ai-hub: ETA placement primitives — relative GPU throughput, the four-term estimate, and an observable breakdown (aicor
- ai-hub: leases go live on the wire — origin fields on submit, /job/<id>/keepalive, /bye, and the reaper that cancels w
- ai-hub: the chat providers move in — fleet qwen, openai, grok, claude/codex/grok CLIs, the responses driver, and the w
- asset-creator: the engine — one pipeline run against the hub, deps-gated, spliced, cancellable, resumable-by-construct
- ai-hub: the machine node mode — --machine binds loopback, registers in ~/.makepad/run, and exits on its own once idle
- asset-creator: makepad-creator-run — the detached client for runs that must outlive a window (aicore §9)
- ai-hub: a native Claude Messages-API provider — API-key or Claude Code OAuth, bounded SSE streaming, injected tools (a
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- asset-creator: the preset tables move in — fifteen chain-policy constants shared by every creator app (aicore §9 / P6)
- makepad_ai is deleted — every backend is a hub pipe, the agent seam lives with its consumers (aicore §14, decided 2026
- ai-hub: loads hold the machine residency election — set_model_state claims on Loaded and publishes the service port (a
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- ai-hub: pick_for_domain_eta — ETA-ranked placement over the shared hard-filter core (aicore §6 / P4)
- asset-creator: the engine picks a provider per stage at dispatch time — a chain's later stages see fresh fleet state (
- ai-hub: the fabric secret gates the service HTTP surface — bearer on everything but /health and the ticketed peer path
- vj: DREAM runs execute in the app — pipelines.rs becomes the run it used to watch (aicore §9 / F1)
- asset-creator: the runner — generate one thing and put it in the catalog, one implementation for every surface (aicore
- chat-ui: the session runs in the app — no broker anywhere on the chat path (aicore P8 / F5)
- asset-store: assets.query is a first-class query endpoint — the bounded SQL surface outlives the broker (aicore P8 / F
- asset-creator: CreatorTools — the chat tool pack for a store that only stores (aicore §9 / P8)
- asset-store: the shrink — the store stores (aicore P7)
- importer + asset-server host: the coordination era ends (aicore P7)
- store config purge + asset-ui goes fleet-direct; the derive protocol gets its route proof (aicore P7)
- client + chat dispatcher: the dead wire comes out (aicore P7/P8)
- ai-hub: 0.3.0 — the health version says which era a node runs
- ai-hub: the default fleet is 'gen' — apps hear the LAN without env plumbing
- ai-hub: the preload note percents the prefill, not the job bar
- ai-hub: conversations keep their KV — the wire mirror, the lane identity, the in-turn dynamic context (aicore §7)
- ai-hub: an open-think model is thinking from its first token
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
- zero-warning sweep, round two — the first full-workspace pass
- zero-warning sweep, round three — the model lanes and the deep examples
- zero-warning sweep, round four — the last stragglers
- zero-warning sweep, round five — vj and chat-ui
- zero-warning sweep, round six — three cascades
Co-authored-by: Claude <info@makepad.nl>
A Turing box (RTX 2080 Ti, sm_75) lost ALL of CUDA because two kernel
files refused to compile for it, and one failed kernel build means the
stub store — surfaced in the VJ as "stems: model error: no compiled-graph
device" on the DJ tab.
diffusion_ops.cu used three sm_80-only pieces unguarded: bf16 wmma
fragments (the type itself is incomplete before Ampere), cp.async, and
the m16n8k16 mma shapes. The cp.async helpers now fall back to
synchronous copies below sm_80 — the f16 wmma flash/sdpa kernels lose
their prefetch overlap on Turing, not their contents — while the bf16
and FA2 kernels are compiled out and their launchers refuse pre-sm_80
devices with cudaErrorNotSupported instead of returning a buffer the
kernel never wrote.
fattn/common.cuh made mkllm_unused_vars constexpr: the no-cp.async
branch of ggml_cuda_fattn_mma_get_nstages calls it, and a non-constexpr
callee poisoned the constexpr config chain on exactly the pre-Ampere
device pass — the arch nobody had compiled for.
Stems verified on the 2080 Ti: stems-ops-check all green (SNR 137-147 dB
against the CPU reference), two tracks separated end to end, output
confirmed clean by ear.
Co-authored-by: vjroger <r.deleeuw@qogni.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The model code was spread across eight crates that had grown into each other:
ggml and cuda and mlx each owned part of a tensor runtime, llama and tts and
voice2 each owned part of a model, and libs/diffusion owned everything else.
They are now one tree with an explicit shape:
libs/ai/cuda — kernels and launch surface
libs/ai/metal — Metal shaders and the shim
libs/ai/llm — the language-model runtime (sessions, lanes, contexts,
the CUDA and Metal executors, the compiled Metal path)
libs/ai/models/ — common, flux, h3, music, paint, speech, stems, vision
libs/diffusion is not deleted but demoted: what remains is the VALIDATOR
crate — several dozen `*_validate.rs` oracles that check a native
implementation against a reference, which is where they belong now that the
implementations live next door.
The functional work inside the move is mostly in the LLM runtime: N lanes that
draft while one verify batch serves all of them, per-slot prefill over a shared
folded attention arena, speculation that survives batching, and a scheduler
that reports rather than publishes. And in the CUDA build: a machine without
usable CUDA must still LINK (and say so), the default kernel arch is the
building machine's GPU, `NO_CUDA` forces the stub even where the toolkit
exists, and kernels compile in parallel with progress.
libs/video_flow is new here: classical optical flow estimation and the `mkfl`
motion-field payload — a flow field measured from a clip without a model,
which is what drives free-rate bounce-looping playback and the uprez/tween
enhance pipe.
Admin
Rogier de Leeuw
