Apps ask the hub for a recognizer or a voice and get one; where it runs is
the hub's decision. AiHub::start_stt / start_tts return poll-driven
sessions shaped like the chat session. The Auto ladder is Whisper/Kokoro in
this process (weights present, machine election), on the machine node over
loopback, on a LAN node, else the OS engine; SpeechReach::Local is the
"don't reach out" knob. Audio always comes back as PCM: the app owns the
device.
Three layers:
- makepad-ai-speech is the whole speech model family, engines only.
libs/voice (Whisper + Silero VAD) folds in as the `whisper` and `vad`
modules next to kokoro and indextts, each a cargo feature; the Apple
bridges and the Speaker/VoiceTranscriber selection leave it.
- makepad-system-speech (new) is the OS speech services as blocking fns:
Apple SpeechAnalyzer/AVSpeechSynthesizer via Swift, Windows.Media.Speech*
on the vendored bindings, Android SpeechRecognizer/TextToSpeech through
MakepadSpeech.java (API 26 floor), espeak-ng on Linux. It models the two
STT shapes honestly: PCM in (Whisper, Apple) versus an engine that owns
the microphone (Android, Windows), with capabilities the caller reads.
- the hub grows speech sessions, in-process Whisper/Kokoro workers with the
residency election, a `whisper` wire backend (stt domain, registry entry
pinned to ggerganov/whisper.cpp) so a Mac can serve a Quest, and a
`language` field on the generate request.
Consumers: the Window voice input runs on an STT session and switches to
engine-mic mode when the recognizer owns the microphone; converse's
SpeechOutput is a lazily started TTS session plus a pump thread; route
drops its private speech copy for converse; vj's lyrics fallback and the
alignment bakes call the engines directly.
Verified here: speech-roundtrip through the real sessions (Apple voice in,
in-process Whisper on Metal out, 4.3% WER); system-speech-test TTS->STT
verbatim; hub/converse/system-speech unit tests; msvc, aarch64-android and
linux-gnu cross-checks; Java against android-34. Windows, Android and Linux
bridges are compile-checked only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Squashed from work:
- rtsmap: one seeded generator for tiled strategy maps
- map: bake a runnable Amsterdam test map from inside the route app
- route: the first-run test map starts itself, and its buttons are guarded
- map: the test map bakes its road faces too, from the same shared pass
- map_tiles: mkmap-extract — the weave is reversible, world-cells reconstructible from world.mkmap
- route: the local dispatcher rides the hub — its copied engine is deleted, limits preserved (aicore P1)
- route + converse: off makepad_ai — the Agent seam moves to converse, route's cloud dispatcher rides the hub's Claude p
- ai-hub: chats run the machine election — route to a serving holder, wait on a loading one, claim and publish when open
- libs: the zero-warning sweep — stitch casts say what they mean, xatlas keeps upstream's surface quietly
Squashed from work; the fine-grained history is under tag archive/work-2026-08-29:
- mp* wave: mpwm window manager + the mp app family, WM API, theme bridge, PDF engine fix
- mpwm polish wave: terminal key focus, focus-history close order, pop-back-to-origin, occupied-workspace cycling, demo
- work: land the sources the last commits reference
- route: the assistant icon the committed UI references
- fast_inflate: the benches name their dev-deps
- gif/weezl: drop the vendored benches nobody can run
- weezl: the decode tests generate their own LZW fixture
- README: a build quick-start for macOS and Windows, the honest Linux
story, what CUDA is for and how to install the separation model.
- AGENTS.md: the `--remote` control surface protocol, so the harness it
documents is usable without reading platform/src/remote.rs.
- Cargo.toml: workspace membership for the crates this series adds and
removes.
- tools/: the Windows box scripts (wincmd, winps and friends, winrun) and
remote_smoke.sh — how a build gets driven on a remote machine.
- apps/asset-server: the standalone server binary and its README.
- Small follow-ups in libs/{windows,apple_sys,makepad_test,mbtile_reader,
converse} and apps/route, plus .gitignore and makepad.splash.
5s mtime watch on the active archive (root.mkidx for mkmap); on change,
Failed placeholders clear and the visible loop re-requests. Workers
already reopen per batch, so an atomic shard-set swap appears live —
apps/route now points at world.mkmap and starts empty until cell-001
(NL) lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Texture-mode passes append to one retained command buffer flushed at the
window pass (safety flush at repaint start for texture-only frames) —
the 12-pass gauss pyramid paid ~1ms commit/schedule latency PER PASS.
Profiling mode (MAKEPAD_GPU_PROFILE) keeps per-pass buffers so spans
stay attributable. Startup window 3400x2050 for pixel-heavy iteration.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Europe re-cut with these buckets happens on explicit go; the NL cut
lives in local/maps (untracked) for cross-session stability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
libs/mbtile_reader gains the .mkmap consumer: root.mkidx parse (Hilbert
tile ids, root ranges -> brotli leaf directories -> shard/offset/len),
positioned shard reads, same get_metadata/get_tile_decoded surface as
MbtilesReader; TileArchiveReader sniffs the path so mbtiles_path can
point at either. Index format v2 drops JSON: the metadata section is now
varint KV like the leaves (writer + reader; no serde in the container).
Loader, zoom-range probe and the headless harness go through the enum;
bridge-dz/overlay sidecars stay mbtiles. Harness parity via shards:
worst AMS rz16 164.7ms, z12 native 51.3ms — identical to mbtiles. App
repointed to local/maps/europe-base-br.mkmap (111 shards + root.mkidx,
the exact bytes a CDN would serve).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Overnight run 2026-08-01: 9,380,390 tiles z2-14 in 3.2h, faces bake 1h
(+8GB, worker-side brotli q10), transmux to 99 shards all <510MB.
App repointed to europe-base-br-faces.mbtiles. nl-bridge-dz verified
byte-identical against the new base (deterministic feature indices).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- The boolean union cascade (59-77% of heavy tile builds) is now baked
per (tile, bucket 14/15/16) as additive field 101: cascade output made
deterministic (1/64 grid snap, canonical ring order, sorted tiers) and
bit-reproducible; renderer consumes guarded by structure signature +
coordinate checksum with runtime fallback; MAKEPAD_NO_BAKED_FACES=1.
Roundtrip bit-identical on all keys x buckets x dz configs. New
tools/map_bake producer (worker-pool incl. worker-side brotli).
- z14 streets merging never ran (early-out at DETAIL_ZOOM): fixed —
1957->1285 ways (-34%), payload -20MB, cascade input shrunk at source.
- Slow-tile log now prints per-stage ms + a ready-to-paste headless repro
command; harness honors archive/detail/dz/overlay/rz/3D envs.
- Worst AMS tile: 2-3s cold -> 175ms rz16 (61ms native); NL bake data
882MB at 60ms threshold; Europe needs threshold 150ms + 2 buckets to
fit shard budget (+2-4GB est).
- App -> nl-base-br3-faces archive; bridge-dz rebake required (joins
fail closed against br3 feature indices).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Voice override ("ok nevermind / stop / wait, do X") and ANY typed
prompt now cancel the in-flight local turn (cancel_prompt -> per-token
stop; worker closes the assistant turn and drops its pending tool
calls). Bare "stop"/"nevermind" just halts + silences TTS; otherwise
the new command is sent immediately. Queueing remains for non-override
follow-ups.
- Greedy-decoding tool loops (identical geo_search spam): an identical
repeated call or >10 tool rounds per prompt gets a corrective error
result instead of execution ("answer with what you have").
- geo_search rejects degenerate queries (>64 chars / >8 words) that
crawl the fuzzy index.
- Append-only context has no recovery once full: at >90% usage after a
turn, the local session restarts fresh (cheap with mmap weights);
transcript stays, conversational memory resets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mid-stream VPIO<->plain unit swap glitched the start of the
assistant's own speech; keep the voice-processing unit armed for the
whole capture (standard+Min ducking) instead. set_echo_cancellation and
the options-rebuild plumbing stay for apps that want the trade.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Idle listening now runs PLAIN capture (no VPIO, zero ducking — music
untouched); the app arms the voice-processing unit only while its own
TTS is audibly playing (+0.8s tail), which is the only window where echo
cancellation matters. Options changes rebuild running input units
(AudioUnitAccess.last_input_options); WindowVoiceInput/VoiceWave gain
set_echo_cancellation; route app polls speech playback at 4Hz.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- bridge-dz baked for the entire Netherlands (17026 tiles, 29MB):
AHN-refined over the 8 Amsterdam sheet pairs, solver-only elsewhere.
Baked as 4 north-south strips with 2-tile overlap — the full-bbox
global solve peaked past 60GB RSS — merged with the new mbtiles-merge
subcommand (later input wins on overlaps, block-major write order).
Route app bridge_dz path ams -> nl.
- audio_unit: the new VoiceInput (VoiceProcessingIO) kind takes the input
setup/handler paths (set_input_handler panicked and aborted the app);
VPIO capture forced mono on every platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ggml: read-only mmap module (unix-gated) + two-region Context (mapped
weights / dirty caches) + segmented Metal buffer binding; llama loads
GGUF weights as file-backed clean pages (jetsam-exempt) with owned-arena
fallback (MAKEPAD_LLAMA_NO_MMAP=1). Route app dirty footprint 12GB -> 4-7GB;
model load becomes lazy page-in; A/B byte-identical on 4B + 9B.
- llama: fix graph-cache keying corruption — a graph keyed wider than the
KV cache corrupted attention for any prefill batch >= 2 (flash op reads
permute-node dims baked at build; view reconfigure never reached the
kernel; masks were written cache-narrow). Masks now always fill the full
graph key width and graphs key by 1024-buckets; reconfigure path removed.
Verified byte-exact vs per-length reference across batch 1/2/8/64/512,
short+long prompts, mmap on/off, plus a two-session concurrency probe.
- voice: passive VoiceWaves no longer register the global audio-input
callback (the invisible caption-bar wave stole mic audio — last
registrant wins — and spawned duplicate whisper workers); whisper back
to F16 default (voice Metal library has no quantized kernels; q5_0
failed every GPU matmul); raw transcripts render immediately.
- route: kokoro TTS voice output (speaker toggle; streams reply sentences,
announces nav maneuvers + arrival) with barge-in — voice activity on the
mic stops playback instantly; dispatcher context 8k -> 32k (hybrid KV is
12/48 layers, ~48KB/token); window caption bar suppressed under studio.
- llama-generate: --max-context/--prefill-batch-size + state fingerprints;
new llama_concurrent_probe bin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
