TL;DR
This paper tackles the challenge of training high-quality neural text-to-speech (TTS) models using readily available but noisy broadcast data (like YouTube streams). They propose a novel approach that combines a music filter to clean the audio with a GST-TTS model enhanced by an auxiliary quality classifier, enabling effective TTS training even with limited clean speech.
What Problem Does This Paper Address?
Training advanced neural text-to-speech (TTS) models typically requires vast amounts of clean, high-quality speech data. While the internet has made speech data more accessible, much of it, particularly from personal broadcasts, is contaminated with background music (BGM). Directly using this noisy data leads to degraded TTS output. Existing methods for noise reduction often create a subtle quality difference between cleaned and truly clean speech, which can still hinder TTS model training. The authors aim to bridge this gap by effectively leveraging broadcast data, which is abundant but imperfect, to train robust TTS systems.
Methods
The proposed method involves two main components:
- Music Filter: Inspired by voice separation techniques, a music filter is trained to remove BGM from noisy speech. This filter operates on the magnitude spectrogram of the audio, predicting a spectral mask to isolate the speech signal. The filter is trained using artificially generated noisy speech mixed with BGM at various signal-to-noise ratios (SNRs).
- GST-TTS with Auxiliary Quality Classifier (AQC): The core TTS model is a GST-TTS architecture, known for its ability to control speech style using Global Style Tokens (GSTs). The novelty here is the addition of an Auxiliary Quality Classifier (AQC). The AQC is trained alongside the main TTS model to predict whether the input reference speech is "clean" or "filtered" (i.e., music-removed). This forces the GST's style embedding vector to focus on representing speech quality rather than other stylistic elements like prosody. During inference, the model uses clean speech as the reference to generate high-quality synthesized speech.
Datasets
- KsponSpeech [15]: Used for training the speaker-independent music filter. This dataset comprises approximately 1,000 hours of spontaneous speech from 2,000 speakers.
- YouTube Studio [16] music samples: 68 license-free music tracks were collected and used to create artificial noisy speech for music filter training.
- Clean speech data: A dataset of approximately 5 hours of speech from a single Korean female speaker was used for training the GST-TTS models. This clean data was artificially mixed with BGM to create noisy speech and then processed by the music filter to create filtered speech.
Key Insights and Ideas
- Quality matters more than just "noise": The paper highlights that the subtle quality difference between music-removed speech and truly clean speech is a significant challenge for TTS training. The AQC directly addresses this by explicitly learning to distinguish these two states.
- AQC guides GST: The auxiliary classifier effectively guides the GST's embedding to prioritize speech quality representation, preventing it from being confused by stylistic variations within the filtered data. This is crucial when clean data is scarce.
- Music filter is effective: The music filter demonstrably reduces BGM, improving speech intelligibility as measured by SER, especially at lower SNRs.
- Limited clean data is manageable: The proposed method shows that even with a small proportion of clean speech (e.g., 0.5 hours out of 5 total hours), the system can achieve near-baseline TTS performance.
Key Results
- Music Filter Performance: The music filter improved PESQ scores significantly across various SNRs and reduced SER for noisy speech, particularly at lower SNRs (0-5 dB).
- TTS Performance (Subjective Evaluation): The proposed GST+MF+Aux. model achieved the highest Mean Opinion Scores (MOS) for both quality and naturalness, reaching scores of 3.92 and 3.85 respectively when trained with 2.5 hours of clean speech. This performance was comparable to a baseline TTS model trained with 5 hours of clean speech.
- Outperforming GST+MF: The GST+MF+Aux. model significantly outperformed the GST+MF model (without the AQC), even when the latter was trained with more clean speech. For instance, with 0.5 hours of clean speech, the proposed model achieved a MOS of 3.05, while GST+MF with 2.5 hours of clean speech only reached 1.84.
- Pronunciation Accuracy (SER): The GST+MF+Aux. model showed a significant reduction in SER compared to the GST+MF model. Even with only 1.5 hours of clean speech, its SER was only slightly higher than the baseline TTS trained with 5 hours of clean speech.
Limitations and Future Work
- Data Imbalance: The paper acknowledges that the performance is still influenced by the ratio of clean to filtered speech. While the AQC helps, more clean data generally leads to better results.
- Distortion at High SNR: The music filter can introduce some distortion at high SNRs, which can negatively impact speech recognizer performance.
- Future Work: The authors suggest that further improvements could be made by exploring more advanced speech enhancement techniques or by incorporating more sophisticated methods for handling data imbalance. The effectiveness of the AQC in guiding the quality embedding is a promising direction for future research in controllable TTS.
Summary updated