TL;DR
This paper introduces a "universal" neural vocoder trained on a massive, multilingual dataset, demonstrating it can generate high-quality speech for unseen speakers and languages without explicit speaker conditioning. This work pushes the boundaries of neural vocoding towards a single model capable of handling diverse speech, significantly outperforming speaker-dependent models in many challenging scenarios.
What Problem Does This Paper Address?
Neural vocoders have revolutionized text-to-speech (TTS) by generating highly natural-sounding speech directly from acoustic features like mel-spectrograms. However, traditional neural vocoders often suffer from a critical limitation: they are highly speaker-dependent. This means a vocoder trained on one speaker (or a small group of speakers) typically performs poorly when asked to synthesize speech for a new, unseen speaker. To achieve good quality, researchers often train separate vocoders for each speaker or explicitly condition the model with speaker embeddings. This approach is not scalable for real-world applications where a vast number of speakers and languages need to be supported. The paper tackles this by asking: can we build a single neural vocoder that generalizes well across a wide range of speakers, languages, and even challenging audio conditions, without needing to know the speaker beforehand?
Methods
The core of this work is a WaveRNN-based neural vocoder, referred to as RNN_MS. The authors adapt the WaveRNN architecture, a powerful autoregressive model known for its high-quality audio generation, by modifying its conditioning network. Crucially, they train this vocoder without any explicit speaker information (like speaker IDs or embeddings). This is a departure from many previous works that relied on such conditioning for speaker adaptation.
The RNN_MS model consists of:
- Autoregressive side: A single forward GRU followed by affine layers and a softmax output to predict 10-bit mu-law audio samples at a 24 kHz sampling rate.
- Conditioning sub-network: A pair of bidirectional GRUs that process mel-spectrograms (extracted with 80 coefficients from 50 Hz to 12 kHz) to guide the audio generation.
The key innovation lies in the training strategy: a single, massive model trained on a diverse dataset.
Datasets
The paper utilizes a combination of internal and publicly available datasets:
- Training Data:
- Internal corpora for initial speaker-dependent (SD) models (US English speakers: 2 female, 1 male).
- A multi-speaker dataset comprising 7 American English speakers (4 female, 2 male, 1 child).
- The "Universal" vocoder was trained on a large dataset of 74 speakers from 17 different languages, totaling approximately 149,134 utterances. This diverse dataset is central to the paper's claim of universality.
- Evaluation Data:
- In-domain speakers: Utterances from speakers present in the training data but not seen during training.
- Out-of-domain speakers: * Neutral speakers from VCTK (English) and NITech Japanese samples database.
- Audiobook speech from the Blizzard2016 development set (studio-quality but out-of-domain in speaker/style).
- Unseen Scenarios: * Varied voice qualities (breathy, pressed).
- Degraded signal quality (noise, reverberation, or both).
- Unseen languages (Ahmaric, Swahili, Wolof) recorded in sub-optimal conditions.
- Singing.
- Non-speech vocalizations (grunts, shouts, etc.).
The evaluation primarily uses the MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) test to subjectively assess naturalness.
Key Insights and Ideas
- Speaker Conditioning is Not Always Necessary: The most significant insight is that a high-quality, speaker-independent neural vocoder can be achieved without explicit speaker embeddings or conditioning. This simplifies the model architecture and opens doors for true universality.
- Data Diversity Trumps Quantity (to an extent): The results suggest that training on a wider variety of speakers and languages (even with fewer utterances per speaker) is more beneficial for generalization than training on more data from a limited set of speakers.
- Universal Vocoder Outperforms SD Models: Even in scenarios where speaker-dependent models could theoretically be used, the universal vocoder often performs comparably or better, especially when dealing with out-of-domain speakers or styles.
- Robustness to Non-Studio Conditions is Challenging: While the universal vocoder shows remarkable generalization, it still struggles with highly degraded audio (heavy noise/reverb) and extreme vocalizations (shouts, grunts), indicating areas for future improvement.
Key Results
- In-domain Speakers: The universal vocoder achieved a 98.5% relative MUSHRA score when synthesizing speech from unseen speakers within the training languages, nearly matching natural speech and performing on par with speaker-dependent models.
- Out-of-domain Speakers (Studio Quality): Similar high performance (98% relative MUSHRA) was observed for out-of-domain speakers and languages when recorded in studio conditions.
- Unseen Scenarios: The universal vocoder significantly outperformed speaker-dependent models in challenging unseen scenarios, with relative MUSHRA scores ranging from 58% to 91.6%. For example, it achieved 91.6% for breathy voice and 89.5% for pressed voice.
- Singing and Non-Speech: The vocoder showed moderate success with singing (72% relative MUSHRA) and non-speech vocalizations (average 48% relative MUSHRA), though performance varied significantly within these categories.
- Degraded Audio: Performance dropped to around 78% for noisy/reverberant conditions and as low as 58% for a combination of both.
Limitations and Future Work
- Limitations:
- The universal vocoder is not yet perfectly robust to highly degraded audio (severe noise, reverberation) or extreme vocalizations like shouts and grunts.
- Generating speech from spectrograms with multiple overlapping speakers remains a challenge.
- While it outperforms SD models, the quality in the most challenging unseen scenarios (e.g., heavy noise + reverb) is still significantly lower than natural speech.
- Future Work:
- The authors suggest that future work should focus on improving training data diversity and model expressiveness to achieve the ultimate goal of a truly universal vocoder.
- Understanding what specific training material best teaches universal generalization is a key direction.
- Developing methods to estimate spectrograms from text for any new speaker, language, or style without needing to train a specific model is the ultimate aim.
Summary updated