makepad/libs/ai/models/speech/build.rs
Admin 97e9572f42 speech: one STT/TTS API on every platform, through the ai-hub
Apps ask the hub for a recognizer or a voice and get one; where it runs is
the hub's decision. AiHub::start_stt / start_tts return poll-driven
sessions shaped like the chat session. The Auto ladder is Whisper/Kokoro in
this process (weights present, machine election), on the machine node over
loopback, on a LAN node, else the OS engine; SpeechReach::Local is the
"don't reach out" knob. Audio always comes back as PCM: the app owns the
device.

Three layers:
- makepad-ai-speech is the whole speech model family, engines only.
  libs/voice (Whisper + Silero VAD) folds in as the `whisper` and `vad`
  modules next to kokoro and indextts, each a cargo feature; the Apple
  bridges and the Speaker/VoiceTranscriber selection leave it.
- makepad-system-speech (new) is the OS speech services as blocking fns:
  Apple SpeechAnalyzer/AVSpeechSynthesizer via Swift, Windows.Media.Speech*
  on the vendored bindings, Android SpeechRecognizer/TextToSpeech through
  MakepadSpeech.java (API 26 floor), espeak-ng on Linux. It models the two
  STT shapes honestly: PCM in (Whisper, Apple) versus an engine that owns
  the microphone (Android, Windows), with capabilities the caller reads.
- the hub grows speech sessions, in-process Whisper/Kokoro workers with the
  residency election, a `whisper` wire backend (stt domain, registry entry
  pinned to ggerganov/whisper.cpp) so a Mac can serve a Quest, and a
  `language` field on the generate request.

Consumers: the Window voice input runs on an STT session and switches to
engine-mic mode when the recognizer owns the microphone; converse's
SpeechOutput is a lazily started TTS session plus a pump thread; route
drops its private speech copy for converse; vj's lyrics fallback and the
alignment bakes call the engines directly.

Verified here: speech-roundtrip through the real sessions (Apple voice in,
in-process Whisper on Metal out, 4.3% WER); system-speech-test TTS->STT
verbatim; hub/converse/system-speech unit tests; msvc, aarch64-android and
linux-gnu cross-checks; Java against android-34. Windows, Android and Linux
bridges are compile-checked only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-01 23:32:35 +02:00

110 lines
3.6 KiB
Rust

use std::env;
use std::fs;
use std::process::Command;
fn main() {
let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
// `makepad-ai-cuda` sets `links = "makepad_ai_cuda"` and emits
// `cargo:kernels=1` only when nvcc actually built and archived its .cu
// objects. Cargo forwards that to us as DEP_MAKEPAD_AI_CUDA_KERNELS, which
// is the only honest signal that `src/cuda/backend.rs` can link against a
// CUDA runtime. Without it the CUDA backend compiles to stubs and
// `src/accel.rs` falls back to Metal (Apple) or the CPU.
println!("cargo:rustc-check-cfg=cfg(makepad_ai_cuda_kernels)");
if env::var("DEP_MAKEPAD_AI_CUDA_KERNELS").as_deref() == Ok("1") {
println!("cargo:rustc-cfg=makepad_ai_cuda_kernels");
}
println!("cargo:rerun-if-env-changed=MAKEPAD_VOICE_METAL_PRECOMPILE");
if target_os == "macos" {
build_whisper_metallib();
}
}
fn build_whisper_metallib() {
let precompile_default = env::var_os("CARGO_FEATURE_METAL_PRECOMPILE").is_some();
let precompile_enabled = env::var("MAKEPAD_VOICE_METAL_PRECOMPILE")
.ok()
.map(|v| {
let v = v.trim().to_ascii_lowercase();
!(v.is_empty() || v == "0" || v == "false" || v == "no" || v == "off")
})
.unwrap_or(precompile_default);
let out_dir = env::var("OUT_DIR").unwrap();
let manifest_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
let ggml_src_dir = format!("{}/src/whisper/metal/ggml", manifest_dir);
let ggml_metal_dir = format!("{}/src/whisper/metal/ggml", manifest_dir);
let metal_src = format!("{}/ggml-metal.metal", ggml_metal_dir);
let common_h = format!("{}/ggml-common.h", ggml_src_dir);
let impl_h = format!("{}/ggml-metal-impl.h", ggml_metal_dir);
println!("cargo:rerun-if-changed={}", metal_src);
println!("cargo:rerun-if-changed={}", common_h);
println!("cargo:rerun-if-changed={}", impl_h);
let _ = fs::create_dir_all(&out_dir);
let air_path = format!("{}/ggml-metal.air", out_dir);
let metallib_path = format!("{}/ggml-default.metallib", out_dir);
if !precompile_enabled {
let _ = fs::write(&metallib_path, []);
println!(
"cargo:rustc-env=MAKEPAD_VOICE_GGML_METALLIB={}",
metallib_path
);
return;
}
let metal_status = Command::new("xcrun")
.args([
"--sdk",
"macosx",
"metal",
"-O3",
"-c",
&metal_src,
"-I",
&ggml_src_dir,
"-I",
&ggml_metal_dir,
"-o",
&air_path,
])
.status();
let ok = metal_status.as_ref().is_ok_and(|s| s.success());
if !ok {
println!("cargo:warning=failed to compile ggml-metal.metal to AIR; runtime source compile will be used");
let _ = fs::write(&metallib_path, []);
println!(
"cargo:rustc-env=MAKEPAD_VOICE_GGML_METALLIB={}",
metallib_path
);
return;
}
let metallib_status = Command::new("xcrun")
.args([
"--sdk",
"macosx",
"metallib",
&air_path,
"-o",
&metallib_path,
])
.status();
let ok = metallib_status.as_ref().is_ok_and(|s| s.success());
if !ok {
println!("cargo:warning=failed to build ggml default metallib; runtime source compile will be used");
let _ = fs::write(&metallib_path, []);
}
println!(
"cargo:rustc-env=MAKEPAD_VOICE_GGML_METALLIB={}",
metallib_path
);
}