Abstract
Modern zero-shot multi-speaker text-to-speech (TTS) systems often require thousands of hours of multi-speaker data to generalize to unseen voices. This paper introduces kNN-TTS, a novel framework that achieves high-quality zero-shot TTS using only a single speaker's training data. By leveraging k-nearest neighbors (kNN) retrieval over self-supervised learning (SSL) speech features, the system can "swap" a source voice for a target voice using a short reference audio clip. Our results show that kNN-TTS achieves competitive naturalness and superior speaker similarity compared to state-of-the-art models trained on up to 1000x more data.
Introduction
Zero-shot TTS aims to synthesize speech in the voice of an unseen speaker given only a short reference sample. While current leading models like XTTS and HierSpeech++ produce impressive results, they are extremely data-intensive, requiring thousands of speakers and tens of thousands of hours of transcribed audio. This high barrier to entry makes developing TTS for low-resource languages or specialized domains nearly impossible.
Furthermore, these systems typically rely on complex speaker encoders to generate embeddings. If these embeddings fail to capture the subtle nuances of a target speaker, the resulting similarity suffers. We propose a simpler, non-parametric approach that treats voice cloning as a retrieval problem, significantly reducing training data requirements while maintaining high performance.
Methodology
The kNN-TTS framework treats voice cloning as a retrieval-augmented generation task. The pipeline consists of four primary components:
- Text-to-SSL Model: A base model (e.g., GlowTTS or GradTTS) is trained to convert input text into SSL features. Crucially, this model is trained on only 24 hours of data from a single speaker (LJSpeech). We utilize WavLM-Large features as the target representation.
- Unit Database: A small database is constructed from a short reference clip (30 seconds to 5 minutes) of the target speaker. This audio is processed through the same SSL encoder to create a set of target feature frames.
- kNN Retrieval: For each frame generated by the Text-to-SSL model, the system performs a k-nearest neighbor search within the target speaker's unit database using cosine distance.
- Interpolation and Vocoding: The source features and retrieved target features are blended using an interpolation parameter $\lambda$. This allows for continuous voice morphing. The final features are converted to a waveform using a pre-trained HiFi-GAN vocoder.
Experimental Results
We evaluated kNN-TTS against several state-of-the-art baselines, including YourTTS, XTTS, and HierSpeech++. Training was conducted solely on the 24-hour LJSpeech dataset, while evaluation used speakers from the LibriSpeech test-clean set.
Performance and Efficiency
As shown in the results table, kNN-TTS variants (GlowkNN-TTS and GradkNN-TTS) achieve remarkable results despite the massive disparity in training data. Notably, GlowkNN-TTS achieved a speaker similarity (SECS) score of 0.72, significantly outperforming XTTS (0.40) and HierSpeech++ (0.67).
In terms of computational efficiency, GlowkNN-TTS is highly optimized, requiring only 0.45 GB of memory—roughly 3x less than HierSpeech++. It also maintains a fast real-time factor (RTF) of 0.24, making it suitable for deployment.
Key Insights
- Feature Proximity: The success of kNN-TTS stems from the observation that SSL representations from different speakers are "linearly close" in feature space. Replacing source frames with their nearest neighbors in the target space effectively transfers speaker identity while preserving phonetic content.
- Data Scaling: While baselines require thousands of hours of data, kNN-TTS achieves competitive naturalness (UTMOS) with just 24 hours of single-speaker data.
- Reference Audio Requirements: Ablation studies indicate that while the model benefits from more reference audio than some "one-shot" baselines, performance stabilizes with approximately 1 minute of target audio.
Discussion and Limitations
While kNN-TTS is highly efficient, it has some limitations. Currently, the prosody and rhythm of the output are primarily determined by the single-speaker source model. If a target speaker has a significantly different speaking rate, the model does not yet adapt its duration predictions. Future work will explore techniques to better match target prosody and expanding the base training set to a small number of diverse speakers to increase the model's expressive range.
Conclusion
kNN-TTS demonstrates that massive multi-speaker datasets are not strictly necessary for high-quality zero-shot TTS. By framing voice cloning as a retrieval task over SSL features, we can achieve state-of-the-art speaker similarity and naturalness with a fraction of the training data. This approach paves the way for accessible, high-fidelity TTS in low-resource scenarios.
Summary updated