Abstract
Zero-shot multi-speaker text-to-speech (TTS) aims to synthesize high-quality speech in a target voice using only a brief reference clip. Traditional methods often suffer from content leakage and poor generalization to unseen speakers. This paper presents a novel "negation" strategy that isolates a speaker's unique vocal signature by mathematically subtracting linguistic content from the total audio signal. Combined with a multi-stream Transformer architecture and Style-Adaptive Layer Normalization (SALN), the proposed model achieves superior voice similarity and lower error rates, particularly for out-of-domain speakers.
Introduction
The primary challenge in zero-shot TTS is disentangling speaker identity (timbre) from linguistic content. Current models frequently experience "content leakage," where the words in the reference clip interfere with the target text, leading to artifacts or unstable speech. Furthermore, generalizing to speakers whose vocal characteristics differ significantly from the training set remains a hurdle. We address these issues by introducing a Content-Information-Free (CIF) speaker embedding derived through feature negation.
Methodology
The proposed architecture consists of three main components designed to improve disentanglement and synthesis quality.
Negated Feature Extraction
The core innovation is the subtraction-based logic for speaker embedding. The model utilizes a pretrained EnCodec to obtain a comprehensive audio representation. Simultaneously, a content extractor uses Instance Normalization to strip away stylistic elements, leaving only linguistic information. By subtracting this content representation from the full EnCodec representation, the model isolates a Content-Information-Free (CIF) speaker embedding—a pure vocal fingerprint.
Multi-stream Transformers and SALN
To capture the complexity of diverse voices, the model employs parallel Transformer streams. This ensemble approach allows different parts of the model to hypothesize various aspects of the speaker's voice, which are then unified via attention pooling. To integrate these features, Style-Adaptive Layer Normalization (SALN) is used to "infuse" speaker characteristics into the text representations at multiple stages of the synthesis process, rather than simple concatenation.
Experimental Setup
The model was trained on the LibriTTS (train-clean-100) dataset, comprising 53 hours of speech from 247 speakers. Audio was resampled to 22,050 Hz, and text was converted to phonemes to ensure a focus on acoustic properties. Evaluation metrics included Mean Opinion Score (MOS) for voice similarity, Mel Cepstral Distortion (MCD) for audio fidelity, and Word Error Rate (WER) to assess content leakage and stability.
Results and Discussion
The proposed model was benchmarked against strong baselines, including Mellotron and Meta-StyleSpeech.
As illustrated in Figure 4, the model consistently outperforms baselines across various scenarios, particularly in unseen speaker categories. Quantitative results confirm this trend: for unseen speakers, the model achieved a MOS of 3.84 (vs. 3.25–3.51 for baselines) and a significantly lower WER of 16.14% (vs. 26.76% for Mellotron). The lower MCD of 8.57 further confirms higher audio fidelity. These results validate that the negation strategy effectively reduces content leakage, leading to more stable and accurate synthesis.
Conclusion
This work demonstrates that "negation" is a powerful tool for feature disentanglement in TTS. By removing linguistic content from the total audio signal, we can isolate a more robust speaker representation. While the current reliance on Instance Normalization for content extraction is a potential limitation, the results on LibriTTS are promising. Future work will explore more complex datasets and real-time optimization.
Summary updated