A Neural Text-to-Speech Model Utilizing Broadcast Data Mixed with Background Music

Hanbin Bae, Jae-Sung Bae, Young-Sun Joo, Young-Ik Kim, Hoon-Young Cho

TL;DR

This paper tackles the challenge of training high-quality neural text-to-speech (TTS) models using readily available but noisy broadcast data (like YouTube streams). They propose a novel approach that combines a music filter to clean the audio with a GST-TTS model enhanced by an auxiliary quality classifier, enabling effective TTS training even with limited clean speech.

What Problem Does This Paper Address?

Training advanced neural text-to-speech (TTS) models typically requires vast amounts of clean, high-quality speech data. While the internet has made speech data more accessible, much of it, particularly from personal broadcasts, is contaminated with background music (BGM). Directly using this noisy data leads to degraded TTS output. Existing methods for noise reduction often create a subtle quality difference between cleaned and truly clean speech, which can still hinder TTS model training. The authors aim to bridge this gap by effectively leveraging broadcast data, which is abundant but imperfect, to train robust TTS systems.

Methods

The proposed method involves two main components:

  • Music Filter: Inspired by voice separation techniques, a music filter is trained to remove BGM from noisy speech. This filter operates on the magnitude spectrogram of the audio, predicting a spectral mask to isolate the speech signal. The filter is trained using artificially generated noisy speech mixed with BGM at various signal-to-noise ratios (SNRs).
  • GST-TTS with Auxiliary Quality Classifier (AQC): The core TTS model is a GST-TTS architecture, known for its ability to control speech style using Global Style Tokens (GSTs). The novelty here is the addition of an Auxiliary Quality Classifier (AQC). The AQC is trained alongside the main TTS model to predict whether the input reference speech is "clean" or "filtered" (i.e., music-removed). This forces the GST's style embedding vector to focus on representing speech quality rather than other stylistic elements like prosody. During inference, the model uses clean speech as the reference to generate high-quality synthesized speech.

Datasets

  • KsponSpeech [15]: Used for training the speaker-independent music filter. This dataset comprises approximately 1,000 hours of spontaneous speech from 2,000 speakers.
  • YouTube Studio [16] music samples: 68 license-free music tracks were collected and used to create artificial noisy speech for music filter training.
  • Clean speech data: A dataset of approximately 5 hours of speech from a single Korean female speaker was used for training the GST-TTS models. This clean data was artificially mixed with BGM to create noisy speech and then processed by the music filter to create filtered speech.

Key Insights and Ideas

  • Quality matters more than just "noise": The paper highlights that the subtle quality difference between music-removed speech and truly clean speech is a significant challenge for TTS training. The AQC directly addresses this by explicitly learning to distinguish these two states.
  • AQC guides GST: The auxiliary classifier effectively guides the GST's embedding to prioritize speech quality representation, preventing it from being confused by stylistic variations within the filtered data. This is crucial when clean data is scarce.
  • Music filter is effective: The music filter demonstrably reduces BGM, improving speech intelligibility as measured by SER, especially at lower SNRs.
  • Limited clean data is manageable: The proposed method shows that even with a small proportion of clean speech (e.g., 0.5 hours out of 5 total hours), the system can achieve near-baseline TTS performance.

Key Results

  • Music Filter Performance: The music filter improved PESQ scores significantly across various SNRs and reduced SER for noisy speech, particularly at lower SNRs (0-5 dB).
  • TTS Performance (Subjective Evaluation): The proposed GST+MF+Aux. model achieved the highest Mean Opinion Scores (MOS) for both quality and naturalness, reaching scores of 3.92 and 3.85 respectively when trained with 2.5 hours of clean speech. This performance was comparable to a baseline TTS model trained with 5 hours of clean speech.
  • Outperforming GST+MF: The GST+MF+Aux. model significantly outperformed the GST+MF model (without the AQC), even when the latter was trained with more clean speech. For instance, with 0.5 hours of clean speech, the proposed model achieved a MOS of 3.05, while GST+MF with 2.5 hours of clean speech only reached 1.84.
  • Pronunciation Accuracy (SER): The GST+MF+Aux. model showed a significant reduction in SER compared to the GST+MF model. Even with only 1.5 hours of clean speech, its SER was only slightly higher than the baseline TTS trained with 5 hours of clean speech.

Limitations and Future Work

  • Data Imbalance: The paper acknowledges that the performance is still influenced by the ratio of clean to filtered speech. While the AQC helps, more clean data generally leads to better results.
  • Distortion at High SNR: The music filter can introduce some distortion at high SNRs, which can negatively impact speech recognizer performance.
  • Future Work: The authors suggest that further improvements could be made by exploring more advanced speech enhancement techniques or by incorporating more sophisticated methods for handling data imbalance. The effectiveness of the AQC in guiding the quality embedding is a promising direction for future research in controllable TTS.

Summary updated

Abstract

Recently, it has become easier to obtain speech data from various media such as the internet or YouTube, but directly utilizing them to train a neural text-to-speech (TTS) model is difficult. The proportion of clean speech is insufficient and the remainder includes background music. Even with the global style token (GST). Therefore, we propose the following method to successfully train an end-to-end TTS model with limited broadcast data. First, the background music is removed from the speech by introducing a music filter. Second, the GST-TTS model with an auxiliary quality classifier is trained with the filtered speech and a small amount of clean speech. In particular, the quality classifier makes the embedding vector of the GST layer focus on representing the speech quality (filtered or clean) of the input speech. The experimental results verified that the proposed method synthesized much more high-quality speech than conventional methods.
BibTeX
@inproceedings{bae2021neural,
  author = {Hanbin Bae and Jae-Sung Bae and Young-Sun Joo and Young-Ik Kim and Hoon-Young Cho},
  title = {{A Neural Text-to-Speech Model Utilizing Broadcast Data Mixed with Background Music}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9415061},
}
Figure