Voice cloning technology has advanced rapidly, moving from complex model fine-tuning to "in-context" learning that can replicate a human voice using just a few seconds of audio. This survey standardizes the terminology of the field and provides a structured roadmap of how modern AI separates a speaker's unique timbre from their prosody.
The Problem: Terminology and Data Scarcity
While the ability to replicate voices has progressed at breakneck speed, the research community has historically struggled with inconsistent definitions. Terms like "few-shot," "zero-shot," and "speaker adaptation" are often used interchangeably, creating a fragmented landscape.
The core challenge remains data scarcity: creating a high-fidelity digital twin of a voice from a tiny snippet of audio. Researchers must also solve the "one-to-many" mapping problem, where the same sentence can be spoken in infinite ways depending on emotion, accent, and intent.
Methods: A Standardized Taxonomy
The survey categorizes voice cloning into four primary paradigms based on the amount of data and the level of model adjustment required:
- Speaker Adaptation: Fine-tuning an entire Text-to-Speech (TTS) model on a specific user's data.
- Few-shot Voice Cloning: A constrained version of adaptation, typically requiring 10 seconds to 5 minutes of audio.
- Zero-shot Voice Cloning: The current "gold standard" where no fine-tuning is required. Models use a Speaker Encoder or an Acoustic Prompt to instantly extract characteristics.
- Multilingual Voice Cloning: Adapting a speaker's voice to a language they may not speak, maintaining identity across linguistic boundaries.
Key Insights and Architectural Shifts
A major architectural shift is underway: moving away from Mel-spectrogram regression toward Neural Codec Language Modeling (NCLM). Models like VALL-E treat TTS as a language modeling task, predicting discrete audio "tokens" similar to how GPT predicts words. This allows for "in-context" speech synthesis, where a 3-second prompt allows the model to "continue" the audio stream in the target speaker's voice without updating internal weights.
Furthermore, the most successful models now focus on disentanglement, effectively separating speech into independent components: content (words), timbre (voice color), and prosody (rhythm and intonation).
Datasets and Performance
Training these data-hungry models requires massive datasets. While VCTK and LibriTTS remain staples for single-language research, newer datasets like Emilia (101,000 hours) are enabling large-scale generation. Specialized sets like IEMOCAP are used to train for emotional states like joy or sadness.
Recent models are approaching human-level naturalness. NaturalSpeech 3 and HierSpeech++ have achieved Mean Opinion Scores (MOS) above 4.0. In terms of likeness, modern zero-shot models can achieve Speaker Embedding Cosine Similarity (SECS) scores above 0.8 with minimal audio, creating a synthetic "fingerprint" mathematically close to the original.
Limitations and Future Directions
Despite these leaps, several hurdles remain:
- Benchmark Gap: The lack of a universal TTS benchmark makes direct comparisons between models difficult.
- Conversational Nuance: Models trained on audiobooks still struggle with the spontaneous nature of real conversation, including filler words and laughter.
- Security and Ethics: As cloning becomes indistinguishable from reality, the authors emphasize the urgent need for robust Deepfake Detection to combat voice-based fraud and misinformation.
Summary updated