Voice Cloning: Comprehensive Survey

Hussam Azzuni, Abdulmotaleb El-SaddikarXiv 2025 · DOI · arXiv

Voice cloning technology has advanced rapidly, moving from complex model fine-tuning to "in-context" learning that can replicate a human voice using just a few seconds of audio. This survey standardizes the terminology of the field and provides a structured roadmap of how modern AI separates a speaker's unique timbre from their prosody.

The Problem: Terminology and Data Scarcity

While the ability to replicate voices has progressed at breakneck speed, the research community has historically struggled with inconsistent definitions. Terms like "few-shot," "zero-shot," and "speaker adaptation" are often used interchangeably, creating a fragmented landscape.

The core challenge remains data scarcity: creating a high-fidelity digital twin of a voice from a tiny snippet of audio. Researchers must also solve the "one-to-many" mapping problem, where the same sentence can be spoken in infinite ways depending on emotion, accent, and intent.

Trend of Voice Cloning-related papers from 2015 to 2024, showing a massive surge in general voice cloning research.

Methods: A Standardized Taxonomy

The survey categorizes voice cloning into four primary paradigms based on the amount of data and the level of model adjustment required:

  1. Speaker Adaptation: Fine-tuning an entire Text-to-Speech (TTS) model on a specific user's data.
  2. Few-shot Voice Cloning: A constrained version of adaptation, typically requiring 10 seconds to 5 minutes of audio.
  3. Zero-shot Voice Cloning: The current "gold standard" where no fine-tuning is required. Models use a Speaker Encoder or an Acoustic Prompt to instantly extract characteristics.
  4. Multilingual Voice Cloning: Adapting a speaker's voice to a language they may not speak, maintaining identity across linguistic boundaries.
Taxonomy of Voice Cloning paradigms and their associated research challenges such as disentanglement and parameter efficiency.

Key Insights and Architectural Shifts

A major architectural shift is underway: moving away from Mel-spectrogram regression toward Neural Codec Language Modeling (NCLM). Models like VALL-E treat TTS as a language modeling task, predicting discrete audio "tokens" similar to how GPT predicts words. This allows for "in-context" speech synthesis, where a 3-second prompt allows the model to "continue" the audio stream in the target speaker's voice without updating internal weights.

Furthermore, the most successful models now focus on disentanglement, effectively separating speech into independent components: content (words), timbre (voice color), and prosody (rhythm and intonation).

Datasets and Performance

Training these data-hungry models requires massive datasets. While VCTK and LibriTTS remain staples for single-language research, newer datasets like Emilia (101,000 hours) are enabling large-scale generation. Specialized sets like IEMOCAP are used to train for emotional states like joy or sadness.

Recent models are approaching human-level naturalness. NaturalSpeech 3 and HierSpeech++ have achieved Mean Opinion Scores (MOS) above 4.0. In terms of likeness, modern zero-shot models can achieve Speaker Embedding Cosine Similarity (SECS) scores above 0.8 with minimal audio, creating a synthetic "fingerprint" mathematically close to the original.

Limitations and Future Directions

Despite these leaps, several hurdles remain:

  • Benchmark Gap: The lack of a universal TTS benchmark makes direct comparisons between models difficult.
  • Conversational Nuance: Models trained on audiobooks still struggle with the spontaneous nature of real conversation, including filler words and laughter.
  • Security and Ethics: As cloning becomes indistinguishable from reality, the authors emphasize the urgent need for robust Deepfake Detection to combat voice-based fraud and misinformation.

Summary updated

BibTeX
@misc{azzuni2025voice,
  author = {Hussam Azzuni and Abdulmotaleb El-Saddik},
  title = {{Voice Cloning: Comprehensive Survey}},
  year = {2025},
  eprint = {2505.00579},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.00579},
}
Figure