Trimmed Closed-Form Test-Time Adaptation of Prosody Predictors from Noisy References

Zero-shot expressive speech synthesis clones a voice from a short reference clip. That clip is also the only evidence of how this speaker, in this scene, shapes pitch and timing. In realistic settings such as movie dubbing, the clip carries music, crowd noise, and effects. A system can learn from the clip in two ways. It can encode it once into a style vector, or it can briefly adapt its prosody model to that clip at inference time. Adaptation at inference is attractive, but it has two costs. It usually needs backpropagation, which is slow. It also trains on "labels" that are really automatic measurements, such as pitch contours from a pitch tracker and phoneme durations from a forced aligner. On noisy audio these measurements are badly wrong in places, with octave jumps, spurious voicing, and misaligned phone boundaries.

We propose to adapt only the small prosody head of a decoupled speech synthesizer, meaning the part that predicts pitch and duration separately from the part that produces timbre and waveform. The head is updated in closed form by regularized least squares, with no backward pass. The update is made robust by a dynamic loss threshold: across a few solve-and-re-score rounds, frames whose pitch or duration residual stays large are dropped, and the cutoff tightens each round. The decoupling makes the adaptation cheap and safe, since timbre and intelligibility are never touched. The dynamic threshold is what makes adaptation from noisy automatic labels work at all.

Core Claim

Backprop-free test-time adaptation has so far been limited to classification. T3A adjusts class prototypes, filtering test samples by entropy. EMO-TTA runs expectation-maximization over Gaussian class models inside a frozen audio-language model. Both need discrete classes. Speech test-time training that handles continuous signals, namely Test-Time Training for Speech and Test-Time Training for Speech Enhancement, uses gradient steps on a shared acoustic encoder with bias-only (BitFit) updates. It applies its self-supervised loss to every frame without any filtering.

Our claim has two parts. First, in a synthesizer whose prosody is decoupled from timbre, the last linear layer of the pitch and duration predictors can be re-fit to a reference clip in closed form. Second, if that re-fit uses a trimmed set of frames chosen by a tightening residual threshold, it recovers the speaker's prosody on held-out sentences under heavy reference noise. Uniform fitting, fixed thresholds, and pitch-tracker confidence weighting do not. The adaptation cost is a small linear solve, not a backward pass.

Concretely, let $h_t \in \mathbb{R}^d$ be the frozen penultimate feature of the prosody predictor for phoneme $t$ of the reference transcript. Let $y_t$ be the pseudo-target, meaning log-F0 averaged over the phoneme or log-duration, extracted from the reference audio. Let $w_0$ be the pretrained last-layer weights. At round $k$ we solve

$$ w_k = \arg\min_w \sum_{t \in S_k} (y_t - w^\top h_t)^2 + \lambda \lVert w - w_0 \rVert^2 , $$

where $S_k$ is the set of kept phonemes and $\lambda$ anchors the solution to the prior. The kept set for the next round is

$$ S_{k+1} = \{\, t : |y_t - w_k^\top h_t| \le \mathrm{med}_k + c_k \,\mathrm{MAD}_k \,\}, $$

where $\mathrm{med}_k$ and $\mathrm{MAD}_k$ are the median and median absolute deviation of current residuals, and $c_k$ decreases over rounds. The first round starts from all frames, and residuals are scored against the prior-anchored fit. Loosening early and tightening later mirrors the schedule of Co-teaching. The cutoff comes from the current residuals themselves rather than from a known noise rate, which is the self-tuning idea in the moving-average threshold of Efficient NLP Model Finetuning via Multistage Data Filtering. Each round is a $d \times d$ solve with $d$ around 256, so its cost is independent of the acoustic decoder's size.

We are candid that this estimator belongs to the family of iteratively trimmed robust regression. The contribution is not the estimator in isolation. It is three things together: using it as backprop-free adaptation for continuous prosody; making it valid by restricting it to a decoupled prosody head; and showing that residual-based trimming beats the domain-specific alternatives a speech engineer would actually reach for.

Research Program

The backbone is a FastSpeech 2–style non-autoregressive synthesizer with separate pitch, energy, and duration predictors. It is pretrained on LibriTTS, and a speaker encoder conditions the acoustic decoder on timbre. This follows the decoupling logic of ProDubber, which pretrains an acoustic model on clean LibriTTS and then freezes it while adapting prosody offline, because updating acoustic weights on noisy expressive clips degraded pronunciation. We move that asymmetry from offline training to inference. Among existing prosody-conditioning methods, CLAPSpeech is the closest. It conditions FastSpeech 2 on contrastively learned, timbre-free prosody representations, but it stays frozen after pretraining. We use a CLAPSpeech-style text-context prosody encoder as an optional feature source for $h_t$.

We will build three things. The first is the adaptation module itself. The same trimmed solve runs per prosody attribute, with pitch and duration trimmed independently, because tracker errors and aligner errors occur in different places.

The second is a controlled corruption protocol that gives ground truth for the pseudo-labels. We take clean ESD references and mix in noise from DNS Challenge 1 at SNRs from 20 dB down to 0 dB, plus music and babble. We then extract pitch and duration from both the clean and the noisy versions. The gap between them defines which pseudo-labels are "wrong," so we can measure the precision and recall of the kept set, just as Delora reports noise-detection precision for text labels.

The third is a gradient counterpart that isolates the latency question. It applies BitFit to the same prosody head, in the style of Test-Time Training for Speech, with small-loss frame filtering across gradient steps.

Baselines

The naive reference point is the frozen zero-shot synthesizer. It conditions on the noisy reference through its style and speaker encoders and adapts nothing. Every other method must improve held-out-sentence prosody over it at matched intelligibility.

The second baseline is untrimmed closed-form adaptation: the same ridge solve on all frames. It shows whether adaptation helps before any robustness is added, and what noisy pseudo-labels do to it.

The third group is the robustness competitors. The key-ingredient ablation replaces the dynamic cutoff with a fixed residual cutoff tuned on a development split; Delora argues that fixed cutoffs fail because the right boundary shifts across samples. We also run a Huber-loss fit, and a soft-weighting fit that weights each frame by the pitch tracker's own voicing or periodicity confidence. The soft-weighting variant is the speech analog of the uncertainty-weighted soft targets in Combating Noise. Tracker confidence is the default signal a practitioner would use, so this is the strongest competitor on the robustness question. We count it as beaten only if dynamic trimming yields lower held-out F0 error at 5 dB and below, across speakers, with non-overlapping bootstrap intervals.

The fourth baseline is the enhancement front-end. We denoise the reference with an off-the-shelf enhancer before extracting pseudo-labels. We run it both with the frozen model and with each adaptation variant.

The fifth baseline is gradient test-time training. It applies BitFit to the prosody head, with and without small-loss filtering. A further variant applies BitFit to the whole acoustic model, which tests whether decoupling matters for timbre and word error rate. All adaptation variants receive the same reference clips, the same features, and the same tuning budget for their hyperparameters ($\lambda$, the schedule for $c_k$, learning rate, and step count), searched on one development speaker set.

For published reference points on the target benchmarks, ProDubber reports a WER of 8.04% on V2C-Animation, against roughly 17% for StyleDubber and V2C-Net. V2C: Visual Voice Cloning reports emotion accuracy of 56.25% and MCD-DTW-SL of 10.05. On ESD, SA-RAS reports a naturalness MOS of 3.827 and a style-similarity MOS of 3.556. These numbers come from different backbones, so we will reproduce the systems where code allows and otherwise treat the figures as context rather than head-to-head targets. For the per-utterance latency of test-time adaptation in speech synthesis, we found no reported result on Lacuna.

Why not just…?

Why not just denoise the reference first? This is a real threat. Enhancement removes stationary noise well and may fix most pitch-tracker failures at moderate SNR. We do not assume it loses. We run enhancement both alone and combined with every adaptation variant. Our claim survives only if trimming still adds held-out prosody accuracy on enhanced references at low SNR and on music-overlapped clips. Enhancers are known to leave artifacts or suppress voiced energy in exactly those conditions, and the adapted TTT enhancer of Test-Time Training for Speech Enhancement was itself trained on unfiltered frames. If enhancement plus untrimmed adaptation matches our method, we will report that the trimming is unnecessary.

Why not just weight frames by the pitch tracker's confidence? Tracker confidence is free and targeted. The case against it is that octave errors under babble noise are often confident, so confidence and correctness come apart when noise is strongest. Combating Noise likewise found that soft weighting alone left error concentrated near its decision boundary. The corruption protocol settles this directly. We measure, against clean-derived labels, how well tracker confidence and our residual trimming each separate correct from incorrect pseudo-labels at each SNR.

Why not just rely on the style encoder or retrieval, with no adaptation? Retrieval systems like SA-RAS run in one forward pass, but they are bounded by the diversity of their style database. A single style vector also compresses a speaker's phoneme-level pitch habits heavily. The frozen zero-shot baseline measures this directly. If adaptation does not beat it on held-out ESD sentences, the whole program fails.

Why not just run BitFit gradient steps, as prior speech TTT does? On a small head, gradient steps may be accurate enough and only moderately slow. The comparison is therefore about the accuracy–latency frontier, not accuracy alone. We report wall-clock adaptation time per utterance alongside error. We also note the step-count and learning-rate fragility documented in Test-Time Training for Speech, which a closed-form solve avoids.

Proposed Experiments

The main testbed is ESD. Its 350 parallel utterances per speaker and emotion let us adapt on a few noisy references and score predicted prosody against real recordings of held-out sentences from the same speaker and emotion. The scores are F0 RMSE and correlation after dynamic time warping, duration error, and MCD. A speech emotion classifier gives emotion accuracy, an ASR model gives WER, and a speaker-verification model gives speaker similarity. We report all of these per SNR.

V2C-Animation is the realistic transfer test. Its references contain genuine soundtrack contamination, and we evaluate WER, emotion accuracy, MCD-DTW-SL, and listening tests. LibriTTS test-clean serves as a no-harm check: at clean conditions, adaptation must not degrade naturalness.

Three results decide the proposal. First, at 5 dB and below, dynamic trimming must beat fixed-cutoff, Huber, and confidence-weighted fits on held-out F0 error, and its kept set must show higher precision against clean-derived labels. Second, adapting only the prosody head must leave WER and speaker similarity statistically unchanged relative to the frozen model, while whole-model BitFit degrades them. That contrast would confirm that decoupling is what makes test-time adaptation safe. Third, the closed-form method must reach the gradient variant's accuracy at a small fraction of its adaptation wall-clock time, measured on the same GPU and CPU with batch size one.

Any one of three outcomes would falsify the central claim: confidence weighting matching our trimming, enhancement alone closing the gap, or untrimmed adaptation failing to beat zero-shot conditioning even on clean references.