Apps ask the hub for a recognizer or a voice and get one; where it runs is the hub's decision. AiHub::start_stt / start_tts return poll-driven sessions shaped like the chat session. The Auto ladder is Whisper/Kokoro in this process (weights present, machine election), on the machine node over loopback, on a LAN node, else the OS engine; SpeechReach::Local is the "don't reach out" knob. Audio always comes back as PCM: the app owns the device. Three layers: - makepad-ai-speech is the whole speech model family, engines only. libs/voice (Whisper + Silero VAD) folds in as the `whisper` and `vad` modules next to kokoro and indextts, each a cargo feature; the Apple bridges and the Speaker/VoiceTranscriber selection leave it. - makepad-system-speech (new) is the OS speech services as blocking fns: Apple SpeechAnalyzer/AVSpeechSynthesizer via Swift, Windows.Media.Speech* on the vendored bindings, Android SpeechRecognizer/TextToSpeech through MakepadSpeech.java (API 26 floor), espeak-ng on Linux. It models the two STT shapes honestly: PCM in (Whisper, Apple) versus an engine that owns the microphone (Android, Windows), with capabilities the caller reads. - the hub grows speech sessions, in-process Whisper/Kokoro workers with the residency election, a `whisper` wire backend (stt domain, registry entry pinned to ggerganov/whisper.cpp) so a Mac can serve a Quest, and a `language` field on the generate request. Consumers: the Window voice input runs on an STT session and switches to engine-mic mode when the recognizer owns the microphone; converse's SpeechOutput is a lazily started TTS session plus a pump thread; route drops its private speech copy for converse; vj's lyrics fallback and the alignment bakes call the engines directly. Verified here: speech-roundtrip through the real sessions (Apple voice in, in-process Whisper on Metal out, 4.3% WER); system-speech-test TTS->STT verbatim; hub/converse/system-speech unit tests; msvc, aarch64-android and linux-gnu cross-checks; Java against android-34. Windows, Android and Linux bridges are compile-checked only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
110 lines
3.6 KiB
Rust
110 lines
3.6 KiB
Rust
use std::env;
|
|
use std::fs;
|
|
use std::process::Command;
|
|
|
|
fn main() {
|
|
let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
|
|
// `makepad-ai-cuda` sets `links = "makepad_ai_cuda"` and emits
|
|
// `cargo:kernels=1` only when nvcc actually built and archived its .cu
|
|
// objects. Cargo forwards that to us as DEP_MAKEPAD_AI_CUDA_KERNELS, which
|
|
// is the only honest signal that `src/cuda/backend.rs` can link against a
|
|
// CUDA runtime. Without it the CUDA backend compiles to stubs and
|
|
// `src/accel.rs` falls back to Metal (Apple) or the CPU.
|
|
println!("cargo:rustc-check-cfg=cfg(makepad_ai_cuda_kernels)");
|
|
if env::var("DEP_MAKEPAD_AI_CUDA_KERNELS").as_deref() == Ok("1") {
|
|
println!("cargo:rustc-cfg=makepad_ai_cuda_kernels");
|
|
}
|
|
println!("cargo:rerun-if-env-changed=MAKEPAD_VOICE_METAL_PRECOMPILE");
|
|
|
|
if target_os == "macos" {
|
|
build_whisper_metallib();
|
|
}
|
|
}
|
|
|
|
fn build_whisper_metallib() {
|
|
let precompile_default = env::var_os("CARGO_FEATURE_METAL_PRECOMPILE").is_some();
|
|
let precompile_enabled = env::var("MAKEPAD_VOICE_METAL_PRECOMPILE")
|
|
.ok()
|
|
.map(|v| {
|
|
let v = v.trim().to_ascii_lowercase();
|
|
!(v.is_empty() || v == "0" || v == "false" || v == "no" || v == "off")
|
|
})
|
|
.unwrap_or(precompile_default);
|
|
|
|
let out_dir = env::var("OUT_DIR").unwrap();
|
|
let manifest_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
|
|
|
|
let ggml_src_dir = format!("{}/src/whisper/metal/ggml", manifest_dir);
|
|
let ggml_metal_dir = format!("{}/src/whisper/metal/ggml", manifest_dir);
|
|
|
|
let metal_src = format!("{}/ggml-metal.metal", ggml_metal_dir);
|
|
let common_h = format!("{}/ggml-common.h", ggml_src_dir);
|
|
let impl_h = format!("{}/ggml-metal-impl.h", ggml_metal_dir);
|
|
|
|
println!("cargo:rerun-if-changed={}", metal_src);
|
|
println!("cargo:rerun-if-changed={}", common_h);
|
|
println!("cargo:rerun-if-changed={}", impl_h);
|
|
|
|
let _ = fs::create_dir_all(&out_dir);
|
|
let air_path = format!("{}/ggml-metal.air", out_dir);
|
|
let metallib_path = format!("{}/ggml-default.metallib", out_dir);
|
|
|
|
if !precompile_enabled {
|
|
let _ = fs::write(&metallib_path, []);
|
|
println!(
|
|
"cargo:rustc-env=MAKEPAD_VOICE_GGML_METALLIB={}",
|
|
metallib_path
|
|
);
|
|
return;
|
|
}
|
|
|
|
let metal_status = Command::new("xcrun")
|
|
.args([
|
|
"--sdk",
|
|
"macosx",
|
|
"metal",
|
|
"-O3",
|
|
"-c",
|
|
&metal_src,
|
|
"-I",
|
|
&ggml_src_dir,
|
|
"-I",
|
|
&ggml_metal_dir,
|
|
"-o",
|
|
&air_path,
|
|
])
|
|
.status();
|
|
|
|
let ok = metal_status.as_ref().is_ok_and(|s| s.success());
|
|
if !ok {
|
|
println!("cargo:warning=failed to compile ggml-metal.metal to AIR; runtime source compile will be used");
|
|
let _ = fs::write(&metallib_path, []);
|
|
println!(
|
|
"cargo:rustc-env=MAKEPAD_VOICE_GGML_METALLIB={}",
|
|
metallib_path
|
|
);
|
|
return;
|
|
}
|
|
|
|
let metallib_status = Command::new("xcrun")
|
|
.args([
|
|
"--sdk",
|
|
"macosx",
|
|
"metallib",
|
|
&air_path,
|
|
"-o",
|
|
&metallib_path,
|
|
])
|
|
.status();
|
|
|
|
let ok = metallib_status.as_ref().is_ok_and(|s| s.success());
|
|
if !ok {
|
|
println!("cargo:warning=failed to build ggml default metallib; runtime source compile will be used");
|
|
let _ = fs::write(&metallib_path, []);
|
|
}
|
|
|
|
println!(
|
|
"cargo:rustc-env=MAKEPAD_VOICE_GGML_METALLIB={}",
|
|
metallib_path
|
|
);
|
|
}
|