Abstract
Developing high-quality Text-to-Speech (TTS) systems for Indic languages is complicated by linguistic diversity and a lack of large-scale, high-quality datasets. This paper describes NVIDIA’s approach to the MMITS-VC 2024 Challenge, where researchers scaled two distinct frameworks—RAD-MMM and P-Flow—to support seven Indic languages. By focusing on attribute disentanglement and efficient flow-based modeling rather than massive autoregressive architectures, the team achieved state-of-the-art results. Their P-Flow system ranked first in the zero-shot track, achieving a Mean Opinion Score (MOS) of 4.4 and demonstrating that high-fidelity voice cloning is possible with as little as three seconds of reference audio.
Introduction
While recent advancements in neural codec autoregressive models have pushed the boundaries of zero-shot voice cloning, these models often require thousands of hours of training data and suffer from slow sampling speeds and robustness issues. For languages with relatively scarce data resources, such as many Indic languages, these requirements pose a significant barrier to entry.
In this work, NVIDIA researchers aimed to build a system capable of both few-shot synthesis (learning a voice from five minutes of data) and zero-shot synthesis (cloning a voice from a three-second prompt) across languages including Hindi, Telugu, and Bengali. The objective was to achieve high naturalness and speaker similarity without the computational overhead typical of large-scale autoregressive models.
Methodology
The researchers deployed two specialized architectures tailored to the specific requirements of the MMITS-VC challenge tracks.
RAD-MMM for Few-Shot Synthesis
For Tracks 1 and 2, the team utilized RAD-MMM, a model designed to disentangle speech attributes. Unlike models that treat speech as a single monolithic signal, RAD-MMM uses deterministic predictors to separately manage fine-grained features such as fundamental frequency (F0) and energy. These features are predicted based on text, accent, and speaker identity. This modularity allows the model to perform cross-lingual synthesis—generating a specific speaker's voice in a language or accent they never provided during training.
P-Flow for Zero-Shot Synthesis
For the zero-shot requirements of Track 3, the researchers employed P-Flow, a flow-based model recognized for its speed and data efficiency. The team modified P-Flow’s decoder and integrated the text pre-processing pipeline from RAD-MMM to extend its reach to seven Indic languages. P-Flow adapts to new speakers by conditioning the synthesis on a brief three-second speech prompt, allowing for rapid adaptation to unseen voices.
Waveform Synthesis
Both architectures utilized HiFi-GAN vocoders to convert internal mel-spectrogram representations into high-fidelity audio waveforms. The researchers noted that fine-tuning these vocoders for specific tracks was essential for maintaining audio quality across different speaker profiles.
Data Strategy and Scaling
The study leveraged a combination of competition-provided data and large-scale public datasets to overcome the limitations of the primary dataset:
- MMITS-VC Challenge Dataset: Included seven languages (Hindi, Telugu, Marathi, Bengali, Chhattisgarhi, English, and Kannada) with two speakers per language.
- External English Data: To improve speaker generalization, the authors incorporated LibriTTS and VCTK. While these are English-only datasets, the diversity of speakers helped the models learn a broader range of vocal characteristics.
- Total Scale: For the zero-shot track, the model was trained on approximately 964 hours of speech from 2,287 distinct speakers.
Key Insights
Accent vs. Language
A critical insight in the RAD-MMM setup was the treatment of language as implicit in the phoneme sequence, while "accent" was treated as a separate attribute capturing regional pronunciation nuances. This distinction proved vital for maintaining naturalness in cross-lingual scenarios where bilingual training data was unavailable.
The Value of Cross-Lingual Transfer
Including English-only data significantly improved the performance of Indic language models. The increased speaker diversity allowed the models to better internalize universal speaker characteristics, which then translated to better zero-shot performance in Indic languages. This suggests that speaker identity and linguistic content can be effectively decoupled in multi-lingual frameworks.
Results
The NVIDIA systems demonstrated superior performance in the MMITS-VC 2024 Challenge:
- Track 3 (Zero-Shot): Ranked 1st overall.
- Naturalness: Achieved a Mean Opinion Score (MOS) of 4.4, indicating high speech quality comparable to human speech.
- Speaker Similarity: Achieved a Speaker MOS (SMOS) of 3.62, proving that a three-second prompt is sufficient for effective voice cloning in a non-autoregressive framework.
- Few-Shot Tracks: RAD-MMM remained highly competitive, proving robust for scenarios where small amounts of target speaker data are available.
Conclusion and Future Work
The success of P-Flow suggests that the industry trend toward massive, slow autoregressive models may not be the only path toward high-quality zero-shot TTS. Efficient, non-autoregressive models can achieve state-of-the-art results with significantly less computational cost.
Future research will focus on addressing the scarcity of native Indic speaker data to further improve accent authenticity. Additionally, the researchers identified a need for more "universal" vocoders that can handle a wider array of languages and speakers without the need for track-specific fine-tuning. Further investigation into the sensitivity of zero-shot models to the quality and emotional tone of the reference prompt is also warranted.
Summary updated