TL;DR
This paper tackles the challenge of extracting high-quality, single-speaker speech segments from noisy, multi-speaker media recordings for training expressive Text-to-Speech (TTS) systems. They compare two architectures, one that first segments audio into speech/non-speech and then diarizes speakers (AS+SD), and another that first diarizes and then segments (SD+AS), finding that the AS+SD approach significantly improves the precision and quality of the extracted speech segments.
What Problem Does This Paper Address?
Traditional TTS systems rely on carefully curated, scripted recordings, which limits the expressivity of synthesized voices. To create more natural and expressive TTS, researchers want to leverage "found" audio data from media like radio broadcasts. However, this data is often imperfect: it can contain multiple speakers, background music, and various types of noise. Before this data can be used to train a TTS voice, it needs to be preprocessed to isolate clean, single-speaker segments. This paper addresses the problem of how to best perform this preprocessing by comparing two different architectural approaches that combine audio segmentation (identifying speech vs. non-speech/noise) and speaker diarization (identifying different speakers within an audio stream).
Methods
The core of the paper lies in comparing two architectures for preprocessing media audio:
- SD+AS (Speech Diarization then Audio Segmentation): This approach first applies a speaker diarizer to the raw audio. The resulting speaker clusters may contain a mix of clean speech, noisy speech, and even music. Subsequently, an audio segmentation module is used to estimate the percentage of clean speech within each cluster, which serves as a quality measure. Clusters are then selected based on both duration and this estimated speech quality.
- AS+SD (Audio Segmentation then Speech Diarization): In this architecture, the audio is first processed by an audio segmenter that labels segments as clean speech, music, speech with noise, or other non-speech sounds. Segments identified as non-clean speech are filtered out. The remaining clean speech audio is then fed into the speaker diarizer. The hypothesis is that the diarizer will perform better on cleaner input. Clusters are then selected primarily based on duration.
Both architectures utilize an open-source speaker diarizer (Shout) based on an iterative segmentation-clustering algorithm with Bayesian Information Criterion (BIC) and Gaussian Mixture Models (GMMs). The audio segmentation module is a non-hierarchical system based on Hidden Markov Models (HMMs), trained to classify audio into five acoustic classes.
Datasets
The study uses a database of 35 radio programs from BBC Radio 4, totaling approximately 659 hours of audio. This data was manually transcribed and annotated at the speaker level. However, information about the presence of music, noise, or low-quality speech was not explicitly labeled, which is a key challenge the system aims to address. The audio segmentation module was initially trained on a subset of Catalan/Spanish broadcast news audio.
Key Insights and Ideas
- Preprocessing is Crucial: The paper clearly demonstrates that attempting to extract speech segments without any preprocessing (i.e., assuming each audio file is from a single speaker) results in very low precision.
- Order Matters: The AS+SD architecture, which filters out non-speech/noisy segments before speaker diarization, consistently outperforms the SD+AS architecture. This suggests that providing cleaner input to the diarizer significantly improves its performance.
- Audio Segmentation as a Quality Indicator: Even in the SD+AS approach, the percentage of clean speech detected by the audio segmenter within a diarized cluster shows a strong correlation with the precision of that cluster. This highlights the utility of audio segmentation for quality estimation.
- Limitations of Diarization Alone: The SD+AS system struggles when the diarizer merges speech from multiple speakers into a single cluster while also including non-speech sounds. In such cases, the audio segmenter might incorrectly label the entire cluster as clean speech, failing to detect its low quality.
- Genre-Specific Challenges: The performance of both systems varies significantly depending on the genre of the radio program. Programs with more speakers, longer durations, or higher levels of background noise and music (e.g., drama-sitcoms) are more challenging than cleaner, studio-recorded content (e.g., weather bulletins).
Key Results
- The AS+SD system achieved significantly higher precision and F0.5 scores across various genres compared to the SD+AS system and the baseline "without preprocessing" approach.
- For example, in the "Total" category, AS+SD achieved a precision of 82.56% and an F0.5 of 0.80, compared to SD+AS's 75.35% precision and 0.77 F0.5.
- The AS+SD system demonstrated an average relative precision improvement of around 2.5% over the SD+AS system.
- The paper shows that for clusters longer than 500 seconds, the AS+SD system produced clusters with precision greater than 88%, whereas the SD+AS system had several clusters with precision below 85%.
Limitations and Future Work
- Multi-Speaker Clusters: A key limitation identified is that the audio segmenter is not able to detect when a cluster contains speech from multiple speakers, even if the speech itself is clean. This can lead to the selection of unsuitable segments for single-speaker TTS training.
- Future Work: The authors plan to use the selected speech segments to train new expressive TTS voices. They also aim to improve the audio segmentation module by exploring features derived from Non-Negative Matrix Factorization (NMF) for speech and non-speech component decomposition, potentially using a semi-supervised approach.
Summary updated