Introduction
Diffusion models have recently emerged as the gold standard for high-fidelity synthetic speech generation, offering superior naturalness compared to traditional GAN or flow-based architectures. However, "speaker adaptation"—the process of fine-tuning a multi-speaker model to a specific target voice using minimal data—remains a significant challenge for diffusion-based systems. Standard fine-tuning is often computationally expensive and requires substantial storage for each new speaker. This research investigates parameter-efficient methods to adapt diffusion-based text-to-speech (TTS) models, specifically focusing on the PriorGrad framework, to achieve high intelligibility and speaker similarity with as few as 10 target utterances.
Methodology
The study explores two primary architectural strategies to enhance the adaptability of the diffusion denoiser:
1. Conditional Layer Normalization (CLN)
The researchers introduced a Conditional Layer Normalization (CLN) block into the standard bidirectional dilated convolutional neural network (CNN) denoiser. Unlike standard normalization, this CLN is conditioned on a concatenation of a speaker embedding and a diffusion step embedding. This allows the model to dynamically adjust its normalization parameters based on both the target speaker's identity and the current stage of the denoising process (i.e., the noise level in the signal).
2. Hybrid Transformer-Diffusion Architecture
To overcome the limitations of adapting the denoiser in isolation, the authors proposed a hybrid architecture. In this setup, a Transformer-based decoder (inspired by AdaSpeech) handles the global speaker characteristics, while the diffusion model acts as a sophisticated "post-net." This synergy allows the Transformer to provide structural guidance while the diffusion model restores the fine-grained spectral texture often lost in over-smoothed Transformer outputs.
Experimental Setup
The source model was pre-trained on approximately 460 hours of clean speech from the LibriTTS dataset (1,151 speakers). For adaptation and evaluation, the researchers used the VCTK dataset, selecting 11 speakers with diverse accents. To test extreme data efficiency, only 10 utterances per target speaker were used for the adaptation process.
Key Results and Discussion
The investigation yielded several critical insights into the behavior of diffusion models during adaptation:
- The Post-Net Advantage: Diffusion models proved more effective at refining speech generated by other components than at generating it from scratch for unseen speakers. The hybrid "AdaDiffDec" model significantly outperformed standalone diffusion adaptation.
- Parameter Efficiency: The proposed adaptation method for the diffusion decoder requires only 0.131M parameters, a fraction of the 1.184M parameters needed for a full AdaSpeech adaptation.
- Quantitative Performance: As shown in the objective metrics, the hybrid model achieved a MOSNet score of 3.02 and a Speaker Similarity (SMOS) of 3.24, surpassing the standalone Transformer baseline (MOS 2.38, SMOS 2.94).
- Step-Dependency: By conditioning on the diffusion step, the model learns to apply speaker-specific adjustments more strongly as the signal becomes cleaner, effectively managing the transition from Gaussian noise to structured speech.
Conclusion
This research demonstrates that while pure diffusion adaptation faces hurdles with intelligibility for out-of-domain data, its integration as a post-processing tool within a hybrid Transformer architecture provides a powerful, parameter-efficient solution for high-quality speaker adaptation. Future work will investigate whether alternative denoiser architectures, such as U-Nets, can further improve the inherent adaptability of diffusion-based TTS systems.
Summary updated