TL;DR
This paper introduces a novel framework for speech synthesis that uses Generative Adversarial Networks (GANs) to combat the "over-smoothing" effect common in neural text-to-speech. By training an acoustic model to "deceive" a discriminator that distinguishes between natural and synthetic speech, the authors produce speech parameters that are statistically much closer to natural human speech, with the Wasserstein GAN (W-GAN) variant emerging as the top performer.
What Problem Does This Paper Address?
In Statistical Parametric Speech Synthesis (SPSS), deep neural networks (DNNs) are trained to map linguistic features (like phonemes) to acoustic parameters (like spectrum and pitch). While effective, these models often suffer from over-smoothing.
Because standard training criteria (like Mean Squared Error) encourage the model to predict the "average" value to minimize error, the resulting speech parameters lose the fine-grained variance and detail found in natural speech. This makes the synthetic voice sound muffled, buzzy, or "robotic." Previous attempts to fix this involved manually adding constraints like Global Variance (GV), but these are often rigid and don't capture the full complexity of human speech.
Methods
The authors propose a GAN-based training framework for acoustic models:
- The Generator: This is the standard acoustic model (a DNN) that predicts speech parameters ($y$) from linguistic features ($x$).
- The Discriminator: A second DNN trained to distinguish between natural speech parameters and those produced by the generator.
- Hybrid Loss Function: Instead of just minimizing the generation error (MGE), the generator is trained on a weighted sum of MGE and an adversarial loss. This adversarial loss forces the generator to produce outputs that the discriminator perceives as "natural."
The researchers experimented with several GAN variants to see which mathematical "distance" between distributions worked best for speech:
- Standard GAN: Minimizes Jensen–Shannon (JS) divergence.
- f-GAN: A generalized framework including KL-divergence.
- Least Squares GAN (LS-GAN): Uses a Pearson $\chi^2$ divergence.
- Wasserstein GAN (W-GAN): Minimizes the Earth-Mover’s distance, which is known for more stable training.
Datasets
The study utilized the ATR Japanese speech database, specifically:
- Speaker: One male speaker.
- Size: 503 phonetically balanced sentences (450 for training, 53 for evaluation).
- Features: 24th-order mel-cepstral coefficients (spectral parameters), $F_0$ (pitch), and 5 band-aperiodicity excitation parameters.
Key Insights and Ideas
- Implicit Statistical Modeling: Unlike previous methods that required researchers to explicitly define features like "Global Variance" or "Modulation Spectrum," the GAN learns these natural statistics implicitly. The discriminator acts as a learned, high-dimensional "naturalness" metric.
- W-GAN is the Winner: The Wasserstein GAN significantly outperformed other variants. The authors suggest that the Earth-Mover’s distance provides a more useful gradient for the generator to follow, especially when the synthetic and natural distributions are very different.
- The "Anti-Spoofing" Connection: The authors point out that the discriminator is essentially a "voice spoofing detector." By training the synthesizer to beat a spoofing detector, they inherently make the speech more realistic.
- Feature Functions: They found that using a simple "Identity" function (passing raw parameters to the discriminator) worked better for improving quality than using more complex "Static & Delta" features often used in anti-spoofing research.
Key Results
- Subjective Quality: In AB preference tests, listeners significantly preferred the GAN-based synthesis over the conventional MGE-trained models for both Text-to-Speech (TTS) and Voice Conversion (VC).
- Over-smoothing Reduction: Visual analysis (scatter plots and GV curves) confirmed that the GAN-generated parameters were as widely distributed as natural speech, whereas conventional models produced tightly clustered, "safe" averages.
- MOS Scores: In a Mean Opinion Score (MOS) test, the W-GAN variant achieved the highest score (~3.5), significantly outperforming the standard GAN and other divergences.
- Versatility: The method proved effective across different architectures, including simple Feed-Forward networks and more complex Long Short-Term Memory (LSTM) networks.
Limitations and Future Work
- Duration Generation: The GAN approach did not significantly improve phoneme or mora duration. The authors suspect the discriminator has a harder time finding "unnaturalness" in timing compared to spectral features.
- Hyper-parameter Sensitivity: The weight ($\omega_D$) balancing the generation loss and adversarial loss requires careful tuning.
- Future Directions: The authors suggest extending this work to direct waveform synthesis (moving beyond parameters) and incorporating linguistic dependencies directly into the discriminator to ensure the "naturalness" is also contextually appropriate.
Summary updated