0287bc9be8
The distill trained sung lyrics with no durations, aligned each step by alignment search on a linear projection of the encoder: on held-out songs that search collapsed (most tokens one frame, a few 50-250), the duration head learned the collapse, and training left the position feature at 0 while rendering fills it, so the voice sang fluent gibberish (held-out WER 86-97%). Now a recorded line is trained on the frames the front end gives its score: - score::score_from_words: the line's timed words (the lyric aligner's spans, which run on over pauses) and its f0 give a score: each word's syllables (score::syllables, maximal-onset-lite) share its sung span, from where the voice starts after the onset consonants to where it stops plus an unvoiced coda, each on the nearest semitone of its median f0; gaps under 120 ms are legato. score::align_segment renders it through score::align, cut or padded to the segment. - data: Kind::SungAligned, a sung-lyric item with its frames per token (u16 after the audio); Store and Item read and write it. - train: Aligned::from_aligned_item / from_lyric_item; speed perturbation moves token boundaries with the audio; on aligned items note attacks are marked where the front end marks them (a syllable's vowel start). - sing_prep lyrics <words dir> <out dir>: segments + timed words (tools/lyric_words.py, from the harvester's alignment.json) to SungAligned shards. - sing_train ac: the lyric pool takes both kinds; --dropout P (Graph::drop: x(1 + sigma eps) on every encoder/decoder residual branch while training); --init W: start from another run's acoustic weights with a fresh optimiser. - sing_eval (feature eval): held-out songs (the trainer's name hash, data::held_out_name) rendered as copy synthesis, teacher-forced on the score frames, and free from the score (Cantor::render, f0 head), transcribed by our recogniser and scored against the lyric; plus the duration head against the aligned lengths. - tools/distill_train.sh: a round that kept no checkpoint no longer breaks the eval loop (nullglob, empty WER). Checked on node 165 (6.4 h, 6917 lines, 10% of songs held out, 40 held-out lines, recordings themselves 12.5% WER against the lyric): the current best distill checkpoint tf 96.4% / free 97.4%, duration correlation -0.01; from scratch on score frames 12k steps tf 50.5% / free 67.5% (small config 58.7% / 66.6%, dropout 0.2 50.3% / 59.4%); from the speech-trained A1 weights with dropout 0.2, 3k steps tf 27.3% / free 38.1%, duration correlation 0.88 (consonants 0.96). cargo test --release -p makepad-ai-sing --lib (14). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
76 lines
4.3 KiB
Bash
76 lines
4.3 KiB
Bash
#!/bin/bash
|
|
# Cantor trained from scratch on the sung-lyric distill (prep/lyrics, growing),
|
|
# in rounds proportional to the data: a round runs when the clean set has grown
|
|
# by half since the last one; its steps scale with the set (about 100 passes);
|
|
# half the lyric segments are speed-perturbed. The acoustic model holds out 10%
|
|
# of the songs; its checkpoints are scored by held-out word error rate and the
|
|
# best is kept (lyric/eval/best.mksing), sung on the eval lines against DSP.
|
|
# start: setsid nohup ~/nv1/distill_train.sh > ~/nv1/lyric/distill.out 2>&1 < /dev/null &
|
|
# stop: touch ~/nv1/lyric/STOP (finishes the current round)
|
|
set -u
|
|
# A round that kept no checkpoints (the trainer failed to start) has nothing to score.
|
|
shopt -s nullglob
|
|
cd ~/nv1
|
|
B=./makepad/target/release
|
|
L=lyric
|
|
log() { echo "$(date +%FT%T) $*" >> $L/run.log; }
|
|
stopped() { test -f $L/STOP; }
|
|
items() { awk -F'\t' '{s+=$4} END {printf "%d", s}' prep/lyrics/harvest.tsv; }
|
|
secs() { awk -F'\t' '{s+=$5} END {printf "%d", s}' prep/lyrics/harvest.tsv; }
|
|
clamp() { v=$1; [ $v -lt $2 ] && v=$2; [ $v -gt $3 ] && v=$3; echo $v; }
|
|
mkdir -p runs/distill-voc runs/distill-ac $L/eval
|
|
GROW=${GROW:-150} # percent of the last round's data that triggers the next
|
|
log "distill training (data-proportional rounds) start: $(items) segments, $(secs) s"
|
|
|
|
( last=0
|
|
while ! stopped; do
|
|
s=$(secs)
|
|
if [ $((s * 100)) -ge $((last * GROW)) ] || [ $last -eq 0 ]; then
|
|
steps=$(clamp $((20 * s / 64)) 500 30000) # about 20 passes over the clean audio
|
|
resume=""; [ -f runs/distill-voc/voc.mksing ] && resume="--resume"
|
|
log "vocoder round: $s s clean, $steps steps"
|
|
$B/sing_train voc --data prep/lyrics --config base $resume --steps $steps --warmup 300 --batch 32 --frames 200 \
|
|
--log 500 --save $steps --workers 8 --out runs/distill-voc >> runs/distill-voc/train.log 2>&1 || log "vocoder round failed"
|
|
last=$s
|
|
fi
|
|
sleep 120
|
|
done ) &
|
|
|
|
( last=0
|
|
while ! stopped; do
|
|
n=$(items)
|
|
if [ $((n * 100)) -ge $((last * GROW)) ] || [ $last -eq 0 ]; then
|
|
steps=$(clamp $((100 * n / 32)) 2000 30000) # about 100 passes over the set
|
|
half=$((steps / 2))
|
|
resume=""; [ -f runs/distill-ac/ac.mksing ] && resume="--resume"
|
|
log "acoustic round: $n segments, $steps steps, speed augmentation on half the segments; checkpoints scored by held-out WER"
|
|
rm -f runs/distill-ac/ac-0*.ema.mksing
|
|
$B/sing_train ac --data prep/lyrics --config base $resume --steps $steps --warmup 200 --batch 32 --speech-frames 1000 \
|
|
--mix lyric=1 --augment 0.5 --holdout 10 --patience 1000000 --log 500 --save $half --keep --workers 8 \
|
|
--out runs/distill-ac >> runs/distill-ac/train.log 2>&1 || log "acoustic round failed"
|
|
last=$n
|
|
# Held-out mel loss bottoms out long before the voice is intelligible: each
|
|
# kept checkpoint is scored by word error rate on the held-out songs, and the
|
|
# best so far is lyric/eval/best.mksing (its WER in best.txt).
|
|
if [ -f runs/distill-voc/voc.ema.mksing ]; then
|
|
for ck in runs/distill-ac/ac-0*.ema.mksing; do
|
|
step=$(basename $ck .ema.mksing | cut -d- -f2 | sed 's/^0*//')
|
|
tag=distill-$step
|
|
$B/sing_train export --ac $ck --voc runs/distill-voc/voc.ema.mksing --out $L/eval/$tag.mksing > /dev/null 2>&1
|
|
$B/voice_eval --whisper models/ggml-large-v3-turbo.bin --cantor $L/eval/$tag.mksing --diag prep/lyrics --held --items 24 --out $L/eval/$tag-held > $L/eval/$tag-held.txt 2>/dev/null
|
|
w=$(grep "^free " $L/eval/$tag-held.txt | awk '{print $2}' | tr -d %)
|
|
[ -n "$w" ] || { log "eval $tag failed"; continue; }
|
|
log "eval $tag ($n segments): held-out WER $(grep -E '^(tf|free) ' $L/eval/$tag-held.txt | awk '{printf "%s %s ", $1, $2}')"
|
|
b=$(cut -d' ' -f1 $L/eval/best.txt 2>/dev/null || echo 101)
|
|
if awk "BEGIN {exit !($w < $b)}"; then
|
|
cp $L/eval/$tag.mksing $L/eval/best.mksing
|
|
echo "$w $tag" > $L/eval/best.txt
|
|
$B/voice_eval --whisper models/ggml-large-v3-turbo.bin --cantor $L/eval/best.mksing --singer 1600 --out $L/eval/best-lines > $L/eval/best-lines.txt 2>&1
|
|
log "new best $tag: held-out free WER $w%; eval lines $(grep -E '^(dsp|cantor) ' $L/eval/best-lines.txt | tr -s ' ' | tr '\n' ';')"
|
|
fi
|
|
done
|
|
fi
|
|
fi
|
|
sleep 120
|
|
done ) &
|
|
wait
|