Explaining Away the Room: Few-Shot Voice Cloning That Models Noise Instead of Removing It

Building a personal synthetic voice usually needs a minute or more of clean studio speech. The people who most need one, such as users with speech impairments who tire quickly (Conformer Parrotron, CoLM-DSR), or anyone recording on a phone, can usually supply only a few seconds of noisy, reverberant audio. Current practice has two options. The first is to clean the audio with a speech-enhancement model, a network trained to strip noise from recordings, and then clone from the result. The second is to clone directly and accept that the voice comes out tinted by the room. Both lose the speaker. Enhancement trained for perceptual quality removes the fine spectral detail that carries identity (VoiceID Loss). Cloning straight from noisy audio teaches the model the noise along with the voice.

We propose a third option. The generative model learns to produce speech in any acoustic environment, controlled by a small environment code. When a new speaker enrolls, we fit two things to their few noisy seconds: a tiny set of speaker parameters and an environment code. At synthesis time we keep the speaker parameters and switch the environment code to "clean." The noise is never removed from the audio. It is explained by a separate variable and then discarded. We build this on a non-autoregressive flow-matching generator. Such a generator produces a whole spectrogram in a few parallel refinement steps rather than one token at a time, so the clean, personalized voice comes out at the speed interactive systems need.

Core Claim

Our central claim is that analysis-by-synthesis factorization breaks the "noisy, tiny enrollment" barrier better than pre-enhancement, feature-level adversarial removal, or noise-augmented zero-shot prompting. Concretely, it means jointly fitting speaker parameters and an environment code on the noisy enrollment, then synthesizing with the environment fixed to clean. We expect the gain to be largest exactly where existing parameter-efficient adapters break down, below roughly ten sentences (AdaSpeech). It should come at no extra inference cost, because the environment branch only changes a conditioning vector.

The model is a conditional flow-matching generator. Flow matching trains a network to predict, at each point between random noise and a target spectrogram, the velocity that moves the sample toward the data. We write this velocity as $v_\theta(x_\tau, \tau \mid c, \phi, e)$. Here $x_\tau$ is the partially generated mel spectrogram at flow time $\tau$, $c$ is the phoneme sequence with durations, $\phi$ are speaker-specific conditional layer-norm scales and shifts, and $e$ is a low-dimensional environment code. Enrollment fits only $\phi$ and $e$ on the noisy clips $y$, using the standard optimal-transport flow-matching loss:

$$\min_{\phi,\, e}\; \mathbb{E}_{\tau, z}\,\big\| v_\theta(x_\tau, \tau \mid c, \phi, e) - (y - z)\big\|^2 + \lambda \|\phi - \phi_0\|^2 .$$

In this objective, $z$ is Gaussian noise, $x_\tau = (1-\tau) z + \tau y$ is the straight-line interpolant, and $\phi_0$ is the zero-shot speaker initialization from a speaker encoder. The weight $\lambda$ controls how far adaptation may drift from that prior. At inference we generate with $e = e_{\text{clean}}$.

The open question, and what makes this a research problem rather than an engineering one, is identifiability. Will the noise actually go into $e$ rather than leaking into $\phi$ when only a few seconds are available? Our design answers this with pretraining that teaches the model what environments look like, so that $e$ is the cheap explanation for noise and $\phi$ is the cheap explanation for timbre.

How This Sits Against Prior Work

Four strands of prior work motivate the design.

Joint systems show that coupling enhancement to the downstream voice task beats a serial cascade. EStarGAN trains a recurrent enhancement front-end jointly with a voice-conversion network, and domain adversarial voice cloning uses a gradient-reversal classifier to make speaker features noise-invariant inside an autoregressive Tacotron-style model. Both still treat noise as something to subtract or suppress. Both are also slow recurrent or autoregressive systems evaluated with ample adaptation data. We instead model noise generatively, at the few-second scale, inside a parallel generator.

The cascade remains the practical default. The THU-HCSI LIMMITS'24 system won its speaker-similarity track by running FullSubNet+ before fine-tuning a non-autoregressive cloner. That is the strongest evidence a reviewer could cite for "just denoise first," and it is why the cascade is our main competitor rather than a straw man.

Parameter-efficient adaptation supplies the method we reuse and also defines the floor we want to break. AdaSpeech adapts only about 5K conditional layer-norm parameters. Residual adapters adapt about 0.1% of a non-autoregressive decoder. Both were evaluated on clean data, and AdaSpeech degrades below ten sentences. We keep their small speaker footprint and add the environment code as the one new degree of freedom.

Parallel generators supply the backbone and expose the failure we target. Voicebox frames synthesis and denoising as infilling with optimal-transport flow matching, about 20 times faster than an autoregressive VALL-E. VoicePrompter shows that single-step flow-matching voice conversion is competitive. Both condition on the prompt in context, however, so a prompt that is noisy throughout tends to carry its acoustics into the output. DualSpeech separates speaker and text guidance, but its reference conditioner mixes environment with timbre. Our environment code is the explicit third axis that none of these provide.

Speculative decoding for autoregressive speech (Raghavan et al.-style tolerance methods, speech speculative decoding) shows that speeding up token-by-token models requires loosened acceptance rules. That evidence constrains us toward a natively parallel generator rather than an accelerated autoregressive one.

Research Program

The first stage is environment-aware pretraining. We train a Voicebox-style flow-matching generator on LibriTTS with phoneme durations from monotonic alignment search (Glow-TTS). Each clean utterance is paired with simulated corrupted copies: additive noise from a training-only noise corpus (MUSAN) and simulated room reverberation at varied signal-to-noise ratios. The target is the corrupted copy. The environment code comes from a small encoder over the corruption residual, the difference between noisy and clean. A learned "clean" code is used whenever the target is uncorrupted. Because the same utterance, and therefore the same speaker, appears under many codes, the model is pushed to place speaker information in the speaker pathway and acoustic conditions in $e$.

The second stage is enrollment. For a new speaker we initialize $\phi$ from a zero-shot speaker encoder and $e$ from the environment encoder applied to non-speech frames of the enrollment. We then optimize both with the objective above for a few hundred steps, with all backbone weights frozen. We also study a constrained variant in which $e$ is restricted to the span of environment codes seen in pretraining. This limits how much speaker information $e$ can absorb and limits how much noise $\phi$ can absorb.

The third stage is inference and its by-products. Synthesis uses 1 to 4 solver steps with $e_{\text{clean}}$ and a standard parallel vocoder (Parallel WaveGAN). The same fitted model can also resynthesize the enrollment itself under $e_{\text{clean}}$, which gives a generative enhancer for free. This is the task-generalization lens: we test whether one factorized model serves cloning, enhancement, and identity preservation, rather than training three separate systems with conflicting goals.

Baselines

The naive reference point is conventional adaptation directly on noisy data. We fit AdaSpeech-style conditional layer-norm parameters to the noisy enrollment on the same frozen backbone, with no environment code. This baseline shows how much noise leaks into a small adapter. For a clean-data reference, AdaSpeech reports a similarity MOS of 3.96 on VCTK with clean one-minute enrollment.

The strongest existing competitor is the modern cascade. We denoise the enrollment with MP-SENet, which reports PESQ 3.50 on VoiceBank+DEMAND with 2.05M parameters, and then run the same speaker-parameter adaptation on the identical backbone. This is the THU-HCSI recipe with a stronger enhancer. It is the most likely system to match ours, because modern enhancers may be gentle enough to preserve identity. Running it on our own backbone separates the effect of factorization from the effect of architecture.

Two further published competitors cover the other main strategies. For zero-shot prompting we use a flow-matching cloner conditioned in context on the noisy prompt, trained with the same corruption augmentation. On clean VCTK, VoicePrompter reports speaker-encoder cosine similarity 0.865 (0.861 at one step). For adversarial removal we reimplement domain adversarial training on our backbone. That work reports MCD 4.18 versus 4.72 for external denoising on MULTI-SPK Mandarin with CHiME-4 noise, so we also run that exact setting to compare against its numbers.

Two ablations isolate the key ingredient. The first fits $\phi$ alone on noisy data with the environment-aware backbone, setting $e$ to the encoder's estimate without optimizing it. The second applies the pretrained environment control with no speaker adaptation at all, using a zero-shot $\phi_0$ and $e_{\text{clean}}$.

All systems share the backbone, pretraining data, vocoder, enrollment clips, number of solver steps, and an equal adaptation-step and hyperparameter-search budget. Beating the cascade means higher speaker similarity to held-out clean recordings of the same speaker at 3 to 10 seconds of noisy enrollment, at equal or better WER and naturalness. For our noisy VCTK+DEMAND protocol we found no reported result on Lacuna, so we will establish all baseline numbers there ourselves.

Why not just…?

Why not just denoise the enrollment with a strong enhancer and adapt as usual? This is the real threat. The evidence against it is old: serial enhancement plus conversion barely improved MCD (9.62 vs 9.66) in EStarGAN, and external denoising lost to adversarial training in the DAT study. Both used weaker enhancers than MP-SENet. VoiceID Loss shows that higher PESQ can coexist with worse identity, which is the mechanism we expect to hurt the cascade. Our experiments settle the question directly by sweeping signal-to-noise ratio and enrollment length. If the MP-SENet cascade matches us everywhere, factorization is not worth its complexity, and we would report that.

Why not just train a zero-shot model on noisy-prompt-to-clean-target pairs and skip adaptation? This is simpler and plausibly strong at high signal-to-noise ratios. Our concern is that, with an ambiguous prompt, a regression-style objective drifts toward an average voice: noise is suppressed, but identity blurs. The zero-shot ablation with $e_{\text{clean}}$ and the augmented in-context baseline measure exactly this. We expect them to close the gap only when enrollment is very clean or very long.

Why not just use a large autoregressive cloner or fine-tune with full LoRA? Autoregressive models pay latency proportional to output length; Voicebox reports being about 20 times faster than VALL-E. Their attention also breaks on noisy references, as observed in the DAT study. Full fine-tuning overfits recording conditions on a minute of data (AdaSpeech). We include an XTTS-based few-shot system in the spirit of BnTTS for latency and similarity, and a LoRA arm with matched steps. This lets us show where higher capacity starts absorbing noise.

Proposed Experiments

The main testbed builds noisy enrollments for held-out VCTK speakers. We corrupt 3, 5, 10, and 30 seconds of their speech with DEMAND noise at 0 to 15 dB and with simulated reverberation. These test noise types are never seen in pretraining. All similarity scores are computed against separate clean recordings of each speaker, so a system cannot score well by copying the noisy prompt.

Metrics are speaker-encoder cosine similarity and similarity MOS; WER from an ASR model; naturalness MOS; MCD where parallel references exist; a noise-leakage score defined as the energy in the output's non-speech frames relative to speech frames; and real-time factor at 1, 2, and 4 solver steps. For cross-lingual and cross-corpus checks, we repeat the MULTI-SPK Mandarin with CHiME-4 setting to compare against the DAT paper's reported MCD of 4.18. We also run the TMHINT noisy voice-conversion setting against EStarGAN's reported 7.85.

Three checks decide the claim. The first is the crossover curve: similarity versus enrollment length for our method, the MP-SENet cascade, and noisy-only adaptation. The idea succeeds if our curve stays flat down to about 5 seconds while the others drop, and fails if the cascade tracks it.

The second is a swap test for identifiability. We take the fitted $\phi$ from a speaker enrolled in environment A and the fitted $e$ from a different speaker in environment B, and synthesize with both. A speaker classifier should still recognize the voice from $\phi$, and an environment classifier should recognize the environment from $e$. If $\phi$ carries environment information, the factorization has failed regardless of headline scores.

The third is the enhancement by-product. We resynthesize VoiceBank+DEMAND test utterances under $e_{\text{clean}}$ and report PESQ honestly against MP-SENet's 3.50. Generative resynthesis will likely score lower on that metric. We pair it with speaker-verification error on the resynthesized audio to test whether the factorized model is the better enhancer when identity, rather than waveform fidelity, is what downstream tasks need.