TL;DR
This paper introduces a novel framework for automatically building expressive synthetic voices by leveraging unsupervised speaker diarization on diverse, real-world speech recordings. The key innovation lies in using diarization to segment and cluster speech segments, enabling the creation of "average" speaking style models without manual annotation, a crucial step towards more natural and versatile TTS systems.
What Problem Does This Paper Address?
Current text-to-speech (TTS) systems often rely on studio-recorded, neutral speech or painstakingly crafted emotional speech. This limits their ability to produce natural-sounding speech in spontaneous or expressive styles. The proliferation of user-generated content and media sharing platforms presents a vast, untapped resource for training more versatile TTS systems. However, extracting useful, style-specific speech data from these heterogeneous sources is challenging due to noise, varying recording conditions, and the presence of multiple speakers. This paper aims to bridge this gap by developing an unsupervised framework that can automatically identify and cluster speech segments belonging to different speaking styles from raw media data, paving the way for more expressive and adaptable TTS.
Methods
The core of the proposed framework is an unsupervised speaker diarization system. This system takes raw audio as input and automatically segments it into segments belonging to different speakers. The key steps involve:
- Voice Activity Detection (VAD): Filtering out non-speech segments.
- Feature Extraction: Using 19 Mel Frequency Cepstrum Coefficients (MFCCs) as acoustic features.
- Segmentation and Agglomerative Clustering: Grouping speech segments into clusters, each representing a "pseudo-speaker."
- Filtering of Clusters: The system aims to create clusters that are both pure (high speaker purity) and sufficiently long to be useful for training.
These generated clusters, or "pseudo-speakers," are then intended to be used to build speaking-style average voice models. These models can then be used to imbue neutral TTS voices with expressivity through techniques like speaker adaptive training. The paper focuses on evaluating the performance of the diarization module for this purpose.
Datasets
The evaluation in this paper is carried out using the C-ORAL-ROM database. This is a multi-language and multi-style corpus containing speech from various formal and informal situations, extracted from media broadcasts and private recordings. For this study, the authors specifically analyzed Spanish formal media styles, including:
- News broadcasts
- Sports commentary
- Meteorological reports
- Reportage
- Talk-shows
- Scientific press
- Interviews
The database presents challenges such as a high number of speakers (124), with some contributing only short utterances, and variability in recording environments.
Key Insights and Ideas
- Unsupervised Diarization as a Foundation: The paper demonstrates that unsupervised speaker diarization can be a powerful tool for automatically segmenting and clustering expressive speech, eliminating the need for manual annotation.
- The Importance of Segment Length: The performance of the diarization system, particularly in terms of recall, significantly improves for clusters longer than 30 seconds. This suggests a minimum duration threshold is crucial for reliable style extraction.
- Speaking Style Matters: The system performs better on speaking styles where speakers are more accustomed to public speaking or have prepared their discourse (e.g., interviews, news). Spontaneous styles like sports commentary and talk shows present more challenges.
- Precision Over Error Rate: For this specific application (building style models), metrics like precision (speaker purity) and recall are more important than traditional diarization error rates (DER). High precision ensures the generated clusters are representative of a single style.
- Speech/Music Segmentation is Beneficial: The authors note that incorporating a speech/music segmentation module could further improve precision, especially for styles like reportage where background music can interfere with diarization.
Key Results
- The speaker diarization system achieved a low DER (average of 12%) across various speaking styles.
- For styles with prepared discourse (interviews, meteorology, news), DER was lower than 5%.
- For more spontaneous styles (reportage, sports, scientific press, talk shows), DER was moderate (between 10% and 25%).
- Recall was consistently higher than 70% for clusters longer than 30 seconds, indicating good coverage of speakers within those segments.
- Precision was generally high (over 85%) for most speaking styles, especially for longer clusters.
- F-measure results showed that clusters longer than 30 seconds achieved recall higher than 75% for all speaking styles.
- The system's performance was weakly affected by SNR or the number of speakers in a session (Pearson correlation coefficients lower than 0.1%).
Limitations and Future Work
- Perceptual Evaluation Needed: The authors acknowledge that while quantitative metrics like precision are good indicators, a perceptual evaluation is necessary to confirm the similarity of the synthetic voices built from these clusters to the original speakers and styles.
- Impact of Spontaneity: The system's performance is still impacted by the spontaneity of the speech. Future work could focus on improving diarization for highly spontaneous styles.
- Speech/Music Segmentation Integration: The paper suggests that integrating a speech/music segmentation module would further enhance the precision of the diarization process, particularly for noisy or mixed-content recordings.
- Robustness to Environmental Conditions: While the C-ORAL-ROM database has variability, further testing on even more challenging and diverse real-world audio data would be beneficial.
Summary updated