Abstract
With recent advances in speech synthesis, synthetic data is becoming a viable alternative to real data for training speech recognition models. However, machine learning with synthetic data is not trivial due to the gap between the synthetic and the real data distributions. Synthetic datasets may contain artifacts that do not exist in real data such as structured noise, content errors, or unrealistic speaking styles. Moreover, the synthesis process may introduce a bias due to uneven sampling of the data manifold. We propose two novel techniques during training to mitigate the problems due to the distribution gap: (i) a rejection sampling algorithm and (ii) using separate batch normalization statistics for the real and the synthetic samples. We show that these methods significantly improve the training of speech recognition models using synthetic data. We evaluate the proposed approach on keyword detection and Automatic Speech Recognition (ASR) tasks, and observe up to 18% and 13% relative error reduction, respectively, compared to naively using the synthetic data.
Introduction
While text-to-speech (TTS) systems have reached high levels of naturalness, their output is not a direct substitute for real human speech in training Automatic Speech Recognition (ASR) models. A persistent "distribution gap" exists between synthetic and real speech. Naively training on synthetic data often causes models to overfit to the specific characteristics of the synthesizer, leading to poor generalization on real-world data.
Characterizing the Distribution Gap
The authors identify four primary regions where synthetic data distributions diverge from the target real data distribution:
- Artifacts: Unrealistic features or noise introduced by the synthesis process.
- Over-sampling: The generator produces an excess of certain "easy" styles.
- Under-sampling: The generator fails to produce enough variety in complex styles.
- Missing Samples: Entire regions of the real data manifold (e.g., specific accents or background environments) are absent from the synthetic output.
The Synt++ Framework
To bridge this gap, Synt++ introduces two key mechanisms: Rejection Sampling and Double Batch Normalization.
Rejection Sampling
Synt++ employs a discriminator to filter synthetic samples. By training a model to distinguish between real and synthetic speech-text pairs, the system can calculate an acceptance probability for each generated sample. Samples that the discriminator identifies as highly likely to be synthetic (often containing artifacts or representing over-sampled regions) are stochastically rejected. This process effectively reshapes the synthetic distribution to more closely align with the real data distribution.
Double Batch Normalization (BN)
Standard Batch Normalization can be detrimental when mixing real and synthetic data because the global statistics (mean and variance) of synthetic data often differ from real data. Synt++ addresses this by maintaining two separate sets of BN statistics during training—one for real data and one for synthetic data. During inference on real speech, the model uses only the statistics derived from real data, preventing "pollution" from synthetic data characteristics.
Experimental Results
The framework was evaluated on LibriSpeech (ASR) and the Speech Commands dataset (Keyword Detection).
- ASR Performance: On the LibriSpeech 960h task, Synt++ achieved a 13% relative error reduction compared to standard synthetic data augmentation. The impact was even more pronounced in low-resource settings; on the LibriSpeech 100h subset, Synt++ helped reduce the Word Error Rate (WER) from 7.7% to 4.0%.
- Keyword Detection: The method yielded up to an 18% relative error reduction in False Accept Rate (FAR) for keywords like "down," "no," and "stop."
- Ablation Studies: Results confirmed that both rejection sampling and Double BN are necessary components. Removing either technique significantly reduced the performance gains, indicating that they address different facets of the distribution gap.
Conclusion
Synt++ demonstrates that imperfect synthetic data can be a powerful asset for ASR training when handled with care. While rejection sampling cannot address the "missing samples" problem—where the generator is fundamentally incapable of producing certain styles—it effectively mitigates the negative impacts of artifacts and sampling bias. As generative models continue to improve, techniques like Synt++ will remain essential for ensuring that synthetic data translates into real-world performance gains.
Summary updated