Abstract
Zero-shot text-to-speech (TTS) synthesis aims to replicate the voice of an unseen speaker using only a brief reference audio sample. Current approaches typically utilize a fixed speaker encoder to produce a single global embedding, which often lacks the capacity to represent the full range of a speaker's prosody and timbre across different linguistic contexts. We introduce ELF (Encoding speaker-specific Latent speech Features), a novel framework that addresses this "prosody mismatch" by discretizing speech features into a speaker-specific codebook. By employing an attention-based fusion mechanism, the text encoder dynamically queries this codebook to retrieve relevant vocal characteristics. Our evaluations demonstrate that ELF significantly improves speaker similarity and speech quality, outperforming state-of-the-art zero-shot models and rivaling multi-speaker models trained on specific target voices.
Introduction
In multi-speaker text-to-speech (TTS) systems, achieving high-fidelity voice cloning for new, unseen speakers remains a significant challenge. Traditional multi-speaker models are constrained to a fixed set of voices and require intensive retraining to incorporate new identities. Conversely, zero-shot models offer flexibility by extracting a speaker embedding from a reference clip. However, these models often suffer from "prosody mismatch"—where the synthesized audio fails to reflect the natural variations in a speaker's voice that depend on the spoken content. A single vector is frequently insufficient to capture the complex distribution of a human voice, leading to unnatural artifacts when the reference audio's emotion or pace differs from the target text.
Methodology
The ELF framework introduces a two-stage approach to capture and utilize speaker-specific vocal nuances.
Speech Feature Encoding Network (SFEN)
The first stage involves a Variational Autoencoder (VAE) designed to reconstruct raw waveforms from mel-spectrograms. Rather than using raw latent outputs, ELF clusters the encoder's features using k-means++ to generate a speaker-specific "codebook." This codebook serves as a discrete library of the speaker's unique vocal building blocks, representing their identity as a distribution rather than a single point.
TTS with Attention-Based Fusion
The second stage integrates these features into a TTS architecture based on VITS. The core innovation is the Attention-Based Fusion mechanism. Instead of globally conditioning the model on a speaker vector, the text encoder uses multi-head attention to "query" the speaker's codebook based on the input phonemes. This allows the model to sample from the speaker's latent space in a content-aware manner, ensuring that the synthesized timbre and prosody are linguistically appropriate.
Experimental Setup
The performance of ELF was evaluated across several standard benchmarks:
- LibriTTS: Used for large-scale English speech synthesis (585 hours, 2,456 speakers).
- VCTK: Used to assess performance across various accents (44 hours, 108 speakers).
- KSS: A Korean dataset used to evaluate cross-lingual capabilities, testing the model's ability to transfer a speaker's identity from Korean to English.
Results and Discussion
ELF demonstrated superior performance compared to both standard multi-speaker VITS models and the popular zero-shot model YourTTS.
Speaker Similarity and Quality
As shown in the table below, ELF achieved a Similarity Mean Opinion Score (SMOS) of 3.85 for unseen speakers, which notably exceeds the performance of the baseline VITS model on seen speakers (3.73). Furthermore, ELF maintained high intelligibility with a low Character Error Rate (CER) of 1.49%.
Key Insights
- Content-Aware Modeling: Linking text to a speaker codebook via attention allows the model to learn that specific vocal textures are associated with certain linguistic patterns.
- Latent Space Organization: The model exhibits a well-organized latent space, capable of linear speaker blending and successful cross-lingual synthesis.
- Reconstruction as Identity: The authors demonstrate that a latent representation sufficient for perfect waveform reconstruction is inherently sufficient for high-quality synthesis of novel sentences.
Conclusion
ELF provides a robust solution for zero-shot TTS by treating speaker identity as a dynamic distribution. While the model benefits from larger reference sets, it remains highly effective with minimal data. Future work may explore reducing the computational overhead of the attention mechanism for edge deployment and extending this discrete-latent approach to other human-centric generative tasks.
Summary updated