SYNT++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition

Ting-Yao Hu, Mohammadreza Armandpour, Ashish Shrivastava, Jen-Hao Rick Chang, Hema Koppula, Oncel Tuzel

TL;DR

Synt++ is a framework designed to make synthetic speech data more effective for training recognition models. By combining a rejection sampling strategy to filter out low-quality samples and separate batch normalization statistics to handle distribution shifts, the authors achieved up to an 18% error reduction in keyword detection and 13% in ASR compared to standard synthetic data training.

What Problem Does This Paper Address?

While speech synthesis (TTS) has become incredibly realistic, using that data to train Automatic Speech Recognition (ASR) models isn't a "plug-and-play" solution. There is a persistent distribution gap between synthetic and real speech.

The authors categorize this gap into four distinct regions:

  1. Artifacts: Synthetic data often contains structured noise or unrealistic speaking styles that don't exist in the real world.
  2. Over-sampling: The generator might produce too much of a specific, easy-to-synthesize style.
  3. Under-sampling: The generator might produce too little of a complex style.
  4. Missing Samples: There are styles (accents, specific environments) the generator simply cannot produce.

If you train a model naively on this "imperfect" data, the model learns the quirks of the synthesizer rather than the nuances of human speech, leading to sub-optimal performance on real-world tasks.

Methods

Synt++ introduces two primary techniques to mitigate the distribution gap:

1. Rejection Sampling

Instead of accepting every sample the generator produces, Synt++ uses a "filter." They train a discriminator to distinguish between real and synthetic speech-text pairs.

  • The Logic: If the discriminator is very sure a sample is synthetic, it likely contains artifacts or belongs to an over-sampled region.
  • The Process: They use the discriminator’s output to calculate an acceptance probability. This stochastically rejects "fake-looking" samples, effectively pushing the synthetic distribution closer to the real one. This addresses artifacts and over/under-sampling, though it cannot fix the "missing samples" problem.

2. Double Batch Normalization (BN) Statistics

Batch Normalization is standard for stabilizing training, but it relies on calculating the mean and variance of the data. When you mix real and synthetic data, these statistics become biased because the synthetic data has different global properties.

  • The Innovation: Synt++ maintains two separate sets of BN statistics during training—one for real data and one for synthetic data.
  • Inference: During testing (on real speech), the model discards the synthetic statistics and only uses the real ones. This ensures the model's internal normalization remains tuned to the target domain.

Datasets

The researchers evaluated Synt++ on two major tasks:

  • ASR: They used LibriSpeech (960h and 100h subsets) for training and evaluation.
  • Keyword Detection: They used the Speech Commands dataset, focusing on keywords like "down," "no," and "stop."
  • Synthesis: Synthetic data was generated using a "Style Equalization" model trained on LibriTTS.

Key Insights and Ideas

  • The "Aha" Moment for BN: The authors realized that even if the features learned from synthetic data are useful, the global statistics (mean/variance) of that data can "pollute" the model's normalization layers, leading to a shift that hurts performance on real data.
  • Discriminator as a Quality Gate: Rather than using a GAN to iteratively improve the generator, they use a discriminator as a post-hoc selection tool. This is computationally efficient and allows them to leverage existing high-quality generators.
  • Synthetic Data for Low-Resource Scenarios: The most dramatic gains were seen when real data was scarce. On the LibriSpeech 100h set, using Synt++ with an expanded text corpus reduced the Word Error Rate (WER) from 7.7% to 4.0%—a massive 48% relative improvement over real-only training.

Key Results

  • ASR Performance: On the full LibriSpeech 960h task, Synt++ achieved a 13% relative error reduction over naively adding synthetic data.
  • Keyword Detection: The method showed up to an 18% relative error reduction in False Accept Rate (FAR).
  • Ablation Success: The ablation studies confirmed that both rejection sampling and Double BN are critical. Removing either one significantly degraded the performance gains, proving they tackle different aspects of the distribution gap.

Limitations and Future Work

  • The "Missing Samples" Wall: The authors admit that rejection sampling cannot solve the problem of missing samples. If a synthesizer cannot produce a Scottish accent, no amount of filtering will help the model learn to recognize it. Future work might involve "filling in" these gaps with more diverse generative priors.
  • Computational Overhead: Rejection sampling requires running a discriminator and potentially discarding many generated samples, which increases the time and compute required to prepare the training set.
  • Generator Dependence: The system's ceiling is still defined by the underlying controllable generative model; as TTS models improve, the "missing samples" region will shrink, but the need for filtering artifacts will likely remain.

Summary updated

Abstract

With recent advances in speech synthesis, synthetic data is becoming a viable alternative to real data for training speech recognition models. However, machine learning with synthetic data is not trivial due to the gap between the synthetic and the real data distributions. Synthetic datasets may contain artifacts that do not exist in real data such as structured noise, content errors, or unrealistic speaking styles. Moreover, the synthesis process may introduce a bias due to uneven sampling of the data manifold. We propose two novel techniques during training to mitigate the problems due to the distribution gap: (i) a rejection sampling algorithm and (ii) using separate batch normalization statistics for the real and the synthetic samples. We show that these methods significantly improve the training of speech recognition models using synthetic data. We evaluate the proposed approach on keyword detection and Automatic Speech Recognition (ASR) tasks, and observe up to 18% and 13% relative error reduction, respectively, compared to naively using the synthetic data.
BibTeX
@inproceedings{hu2022synt,
  author = {Ting-Yao Hu and Mohammadreza Armandpour and Ashish Shrivastava and Jen-Hao Rick Chang and Hema Koppula and Oncel Tuzel},
  title = {{SYNT++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=k71E1MSA1L2},
}

Loading related content...

Figure