Training Universal Vocoders with Feature Smoothing-Based Augmentation Methods for High-Quality TTS Systems

Jeongmin Liu, Eunwoo SongarXiv 2024 · arXiv · OpenReview

Abstract

While universal vocoders have achieved proficient waveform generation across diverse voices, their integration into text-to-speech (TTS) tasks often results in degraded synthetic quality. To address this challenge, we present a novel augmentation technique for training universal vocoders. Our training scheme randomly applies linear smoothing filters to input acoustic features, facilitating vocoder generalization across a wide range of smoothings. It significantly mitigates the training-inference mismatch, enhancing the naturalness of synthetic output even when the acoustic model produces overly smoothed features. Notably, our method is applicable to any vocoder without requiring architectural modifications or dependencies on specific acoustic models. The experimental results validate the superiority of our vocoder over conventional methods, achieving 11.99% and 12.05% improvements in mean opinion scores when integrated with Tacotron 2 and FastSpeech 2 TTS acoustic models, respectively.

Introduction

In a standard text-to-speech (TTS) pipeline, an acoustic model converts text into acoustic features (such as mel-spectrograms), and a vocoder subsequently transforms these features into an audio waveform. The "Universal Vocoder" is a highly sought-after component capable of synthesizing high-quality speech across various speakers and languages without model-specific adjustments.

However, universal vocoders often suffer from a training-inference mismatch. They are typically trained on "ground-truth" spectrograms extracted from real speech, which are sharp and detailed. In contrast, TTS acoustic models frequently produce "overly smoothed" spectrograms—essentially a blurred version of reality. When a vocoder trained on sharp features encounters these smoothed inputs during inference, the resulting audio quality degrades significantly. While fine-tuning vocoders for specific acoustic models can mitigate this, it destroys their universality and requires expensive retraining for every new voice or model.

Methodology

Smoothing Augmentation Strategy

To address the mismatch, we propose a novel feature smoothing-based augmentation strategy. Instead of training exclusively on pristine ground-truth features, we intentionally introduce varying levels of "blurriness" during the training phase.

Comparison of standard training (a) and the proposed smoothing augmentation training (b).

As illustrated in the figure above, we apply a random 2D triangular low-pass filter (LPF) to the input features during training. The filter sizes for the time ($l_t$) and frequency ($l_f$) axes are randomly sampled for every training step. This randomization ensures the vocoder learns to reconstruct high-quality audio from a wide spectrum of feature qualities, effectively mimicking the errors typically produced by models like Tacotron 2 or FastSpeech 2.

eUnivNet Architecture

Alongside the augmentation strategy, we introduced eUnivNet, an enhanced version of the UnivNet model. Key architectural improvements include:

  • Harmonic-Noise (HN) Model: Helps the generator better distinguish between periodic vocal components and aperiodic noise.
  • Advanced Discriminators: Utilizes Multi-Scale STFT (MS-STFT) and Collaborative Multi-Band (CoMB) discriminators to capture complex audio attributes across different frequency bands.
  • Snake Activation: Replaces standard activation functions with the "Snake" function, specifically designed to help neural networks learn periodic signals like audio waves.
  • Efficiency: Despite these improvements, eUnivNet uses fewer convolutional channels (16 vs 32) than the original UnivNet, prioritizing architectural intelligence over raw parameter count.

Key Insights and Results

The core innovation lies in the realization that a vocoder does not need to see the exact errors of a specific TTS model to fix them. By exposing the model to a diverse range of random smoothing levels, it becomes robust enough to handle the specific smoothing patterns of almost any acoustic model.

Normalized histograms of Mel-spectral distances for different smoothing parameters compared to Tacotron 2.

As shown in the histogram analysis, varying the smoothing parameters ($l_t, l_f$) allows the training distribution to cover the actual error distribution of models like Tacotron 2. This "distribution matching" is what enables high-quality synthesis without model-specific fine-tuning.

Performance Metrics

Evaluations conducted on Korean speakers (both seen and unseen during training) confirmed the effectiveness of the approach:

  • Naturalness Gains: When paired with Tacotron 2, the Mean Opinion Score (MOS) rose from 3.67 to 4.11 (an 11.99% improvement). With FastSpeech 2, MOS increased from 3.07 to 3.44 (a 12.05% improvement).
  • Generalization: The model outperformed baselines for unseen speakers by a wide margin (3.76 vs 3.29 MOS).
  • Speed: eUnivNet generates waveforms 2.54x faster than real-time on a single CPU core, making it suitable for production environments.

Conclusion

Feature smoothing-based augmentation provides a robust, architecture-agnostic solution to the training-inference mismatch in TTS systems. By teaching universal vocoders to handle a spectrum of feature qualities, we achieve high-quality, natural-sounding speech across diverse models and speakers without the need for costly fine-tuning. This methodology holds significant potential for other audio generation tasks, such as singing voice synthesis or music generation, where similar over-smoothing issues occur.

Summary updated

Abstract

While universal vocoders have achieved proficient waveform generation across diverse voices, their integration into text-to-speech (TTS) tasks often results in degraded synthetic quality. To address this challenge, we present a novel augmentation technique for training universal vocoders. Our training scheme randomly applies linear smoothing filters to input acoustic features, facilitating vocoder generalization across a wide range of smoothings. It significantly mitigates the training-inference mismatch, enhancing the naturalness of synthetic output even when the acoustic model produces overly smoothed features. Notably, our method is applicable to any vocoder without requiring architectural modifications or dependencies on specific acoustic models. The experimental results validate the superiority of our vocoder over conventional methods, achieving 11.99% and 12.05% improvements in mean opinion scores when integrated with Tacotron 2 and FastSpeech 2 TTS acoustic models, respectively.
BibTeX
@misc{liu2024training,
  author = {Jeongmin Liu and Eunwoo Song},
  title = {{Training Universal Vocoders with Feature Smoothing-Based Augmentation Methods for High-Quality TTS Systems}},
  year = {2024},
  eprint = {2409.02517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.02517},
}

Loading related content...

Figure