makepad

History

Admin c83ce3617d ai: hub stems and word timing, GPU yield, quantised image models, fleet discovery Stems take any audio file and can answer Ogg stems; speech-to-text answers word-timed transcription and batched forced alignment. A generation the full model cannot fit next to the UI runs on the family's q4 variant; a chat during a GPU yield is answered; a GPU failure fails the job instead of a silent CPU fallback. Block-quantised image transformer on the GPU and a 24 GB-card image model entry. The fleet beacon goes to every interface's broadcast and a hub on the gateway is found where no beacon reaches; the LAN listener can be stopped. Tokenizer and G2P lexicon tables download with their models instead of living in git. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>		2026-10-14 07:24:22 +02:00
..
src	ai: hub stems and word timing, GPU yield, quantised image models, fleet discovery	2026-10-14 07:24:22 +02:00
tests	ai: the AI hub gives way to the UI, smaller models, vision and speech	2026-10-13 12:26:21 +02:00
tools	ai sing: rounds and lyric words can leave a music model's songs out -- lyric_words.py --exclude MODEL skips song groups that model made (from the producer's request in the inbox), and --count-secs prints the harvested clean seconds of the groups kept. lyric_rounds.sh takes EXCLUDE (counted toward its trigger too) and --now. It also reads the summary rows of sing_eval anchored, because the per-line rows matched too and the first round found no scored checkpoint. Checked on node 165: without ace-step-1.5-xl, 17644 of 27745 harvested s and 4.29 h aligned. Round 2 (no ace-step, 6k steps from A1), its best step 4000 against the current best: eval lines WER 20.0% vs 20.3%, pitch 2.90 vs 2.60 c, so the current best was kept.	2026-10-02 19:38:05 +02:00
Cargo.toml	ai: the AI hub gives way to the UI, smaller models, vision and speech	2026-10-13 12:26:21 +02:00
CREDITS.md	ai sing: Cantor, a neural singing voice -- score notes + syllables are aligned to 10 ms frames (the vowel owns the note, onsets lead the beat, codas close it, melismas carry the vowel) with a rule pitch curve (legato spring, scoop, delayed vibrato, drift); an acoustic model (phoneme transformer with a duration head, note/f0 features, an f0 head, a conformer decoder to a 128-bin 48 kHz log-mel, a shallow rectified-flow refiner) and a harmonic-plus-noise source-filter vocoder (every harmonic of f0 from a DSP source, gains and phase per STFT bin from a frame-rate ConvNeXt, iSTFT) so the output pitch is the f0 curve by construction. One model definition for inference and training: nn.rs is a small tape-autograd tensor library with a CPU backend (the reference; matmuls through the Metal/cuBLAS gateway) and nn_gpu.rs a fully device-resident CUDA backend over the new ai-cuda training kernels; batches are items of seg rows (convolutions, RoPE, attention and the STFT family stay inside an item). train.rs has the vocoder loss (3-resolution log-STFT + mel L1, band-limited per item), the acoustic loss (mel, duration, f0 residual, voicing, flow, and monotonic alignment on token means for speech without durations), AdamW with clipping and EMA on host or device, MKSING checkpoints, and a synthetic singer for T0. Bins: sing_render (a sung line, random or trained weights), sing_train (check \| t0 \| voc \| ac), sing_prep (dataset shards). CREDITS.md records the training data (VocalSet, LibriTTS; CC BY 4.0) with checksums. Design: local/agent_state/edits/design/NEURAL-VOICE.md. Checked: cargo test --release -p makepad-ai-sing (11: FFT, STFT/iSTFT inverse and adjoints, finite-difference gradients of every op incl. batched attention and the audio ops, alignment, weights, shards); sing_render on random base weights (22.4 M params) renders 5.7 s at 9.2x real time on this Mac with 2.4 cents pitch error; sing_train check on node 165 (CPU vs GPU losses and gradients within 3.9e-4 exact f32); T0 base run on node 165 (both models' losses fall; vocoder copy f0 error 1.2 cents).	2026-09-30 22:51:40 +02:00