Apps ask the hub for a recognizer or a voice and get one; where it runs is
the hub's decision. AiHub::start_stt / start_tts return poll-driven
sessions shaped like the chat session. The Auto ladder is Whisper/Kokoro in
this process (weights present, machine election), on the machine node over
loopback, on a LAN node, else the OS engine; SpeechReach::Local is the
"don't reach out" knob. Audio always comes back as PCM: the app owns the
device.
Three layers:
- makepad-ai-speech is the whole speech model family, engines only.
libs/voice (Whisper + Silero VAD) folds in as the `whisper` and `vad`
modules next to kokoro and indextts, each a cargo feature; the Apple
bridges and the Speaker/VoiceTranscriber selection leave it.
- makepad-system-speech (new) is the OS speech services as blocking fns:
Apple SpeechAnalyzer/AVSpeechSynthesizer via Swift, Windows.Media.Speech*
on the vendored bindings, Android SpeechRecognizer/TextToSpeech through
MakepadSpeech.java (API 26 floor), espeak-ng on Linux. It models the two
STT shapes honestly: PCM in (Whisper, Apple) versus an engine that owns
the microphone (Android, Windows), with capabilities the caller reads.
- the hub grows speech sessions, in-process Whisper/Kokoro workers with the
residency election, a `whisper` wire backend (stt domain, registry entry
pinned to ggerganov/whisper.cpp) so a Mac can serve a Quest, and a
`language` field on the generate request.
Consumers: the Window voice input runs on an STT session and switches to
engine-mic mode when the recognizer owns the microphone; converse's
SpeechOutput is a lazily started TTS session plus a pump thread; route
drops its private speech copy for converse; vj's lyrics fallback and the
alignment bakes call the engines directly.
Verified here: speech-roundtrip through the real sessions (Apple voice in,
in-process Whisper on Metal out, 4.3% WER); system-speech-test TTS->STT
verbatim; hub/converse/system-speech unit tests; msvc, aarch64-android and
linux-gnu cross-checks; Java against android-34. Windows, Android and Linux
bridges are compile-checked only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Squashed from work:
- rtsmap: one seeded generator for tiled strategy maps
- map: bake a runnable Amsterdam test map from inside the route app
- route: the first-run test map starts itself, and its buttons are guarded
- map: the test map bakes its road faces too, from the same shared pass
- map_tiles: mkmap-extract — the weave is reversible, world-cells reconstructible from world.mkmap
- route: the local dispatcher rides the hub — its copied engine is deleted, limits preserved (aicore P1)
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- mpwm polish wave: terminal key focus, focus-history close order, pop-back-to-origin, occupied-workspace cycling, demo
- work: land the sources the last commits reference
- route: the assistant icon the committed UI references
- fast_inflate: the benches name their dev-deps
- gif/weezl: drop the vendored benches nobody can run
- weezl: the decode tests generate their own LZW fixture
- README: a build quick-start for macOS and Windows, the honest Linux
story, what CUDA is for and how to install the separation model.
- AGENTS.md: the `--remote` control surface protocol, so the harness it
documents is usable without reading platform/src/remote.rs.
- Cargo.toml: workspace membership for the crates this series adds and
removes.
- tools/: the Windows box scripts (wincmd, winps and friends, winrun) and
remote_smoke.sh — how a build gets driven on a remote machine.
- apps/asset-server: the standalone server binary and its README.
- Small follow-ups in libs/{windows,apple_sys,makepad_test,mbtile_reader,
converse} and apps/route, plus .gitignore and makepad.splash.
- ggml: read-only mmap module (unix-gated) + two-region Context (mapped
weights / dirty caches) + segmented Metal buffer binding; llama loads
GGUF weights as file-backed clean pages (jetsam-exempt) with owned-arena
fallback (MAKEPAD_LLAMA_NO_MMAP=1). Route app dirty footprint 12GB -> 4-7GB;
model load becomes lazy page-in; A/B byte-identical on 4B + 9B.
- llama: fix graph-cache keying corruption — a graph keyed wider than the
KV cache corrupted attention for any prefill batch >= 2 (flash op reads
permute-node dims baked at build; view reconfigure never reached the
kernel; masks were written cache-narrow). Masks now always fill the full
graph key width and graphs key by 1024-buckets; reconfigure path removed.
Verified byte-exact vs per-length reference across batch 1/2/8/64/512,
short+long prompts, mmap on/off, plus a two-session concurrency probe.
- voice: passive VoiceWaves no longer register the global audio-input
callback (the invisible caption-bar wave stole mic audio — last
registrant wins — and spawned duplicate whisper workers); whisper back
to F16 default (voice Metal library has no quantized kernels; q5_0
failed every GPU matmul); raw transcripts render immediately.
- route: kokoro TTS voice output (speaker toggle; streams reply sentences,
announces nav maneuvers + arrival) with barge-in — voice activity on the
mic stops playback instantly; dispatcher context 8k -> 32k (hybrid KV is
12/48 layers, ~48KB/token); window caption bar suppressed under studio.
- llama-generate: --max-context/--prefill-batch-size + state fingerprints;
new llama_concurrent_probe bin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
