TL;DR
This paper tackles the challenge of synthesizing natural-sounding spontaneous speech by leveraging "found data" – specifically, conversational podcasts. They introduce a novel pipeline for automatically transcribing and segmenting this messy data, enabling the creation of high-quality synthetic voices that outperform traditional read-speech synthesis for conversational contexts.
What Problem Does This Paper Address?
Creating truly conversational speech synthesis is a significant hurdle. Unlike the controlled environment of read-speech (like audiobooks), spontaneous speech is rife with disfluencies (hesitations, filler words), variable pacing, and unconventional sentence structures. While previous work has explored spontaneous speech, it often relied on small, meticulously hand-annotated lab recordings. The authors aim to bridge this gap by using readily available, large-scale "found data" – specifically, conversational podcasts – which are more representative of real-world speech but present considerable transcription and segmentation challenges. They want to see if high-quality synthesis is achievable from this messier, but more abundant, data source.
Methods
The core of their approach involves several key steps:
- Speaker-Dependent Breath Detection: To segment continuous podcast recordings into manageable utterances, they employ a CNN-LSTM model trained to detect breath events. This is crucial because breath patterns are linked to speech planning and turn-taking in spontaneous conversations.
- Automated Transcription: Instead of manual annotation, they use Google Cloud Speech API for initial transcription, enhanced with "phrase hints" for domain-specific jargon.
- Filled Pause (FP) Identification: Recognizing that FPs (like "uh" and "um") are characteristic of spontaneous speech, they use IBM Watson Speech to Text to identify these, and then integrate them back into the Google API transcript. This allows them to control for FPs during synthesis and evaluate their impact.
- Tacotron 2 Architecture: For the actual speech synthesis, they utilize the Tacotron 2 neural network architecture, a standard for state-of-the-art text-to-speech systems.
- Transfer Learning: To improve pronunciation accuracy, they leverage transfer learning, pre-training models on a large read-speech corpus (LJSpeech) before fine-tuning on their podcast data.
Datasets
- Training/Synthesis Data: The primary dataset is the "ThinkComputers" podcast, from which they extracted approximately 9 hours of clean, single-speaker audio. This is referred to as the TCC (ThinkComputers Corpus).
- Evaluation Data:
- Harvard Sentences: 400 phonetically balanced sentences were synthesized for pronunciation and conversational feature evaluations.
- Arctic Corpus: Used for read speech prompts in the MUSHRA-like test.
- Lab-recorded spontaneous speech: A corpus from a previous study was used for comparison.
- Public domain audiobooks: Used for read speech prompts.
Key Insights and Ideas
- Found Data is Viable: The paper demonstrates that high-quality conversational speech synthesis is achievable from "found data" like podcasts, overcoming the limitations of smaller, manually curated corpora.
- Automated Transcription is Key: The success hinges on robust automated transcription and segmentation techniques, making large-scale spontaneous speech data accessible for TTS.
- Filled Pauses: Authenticity vs. Fluency: While removing filled pauses can lead to a more fluent-sounding voice (fewer deleted words, repeated syllables), the presence of FPs can significantly enhance perceived speaker authenticity.
- Genre Appropriateness: Voices trained on spontaneous conversational data are demonstrably more appropriate for public speaking and casual conversation genres compared to voices trained solely on read speech.
Key Results
- Pronunciation Accuracy: A voice trained with phoneme-level input and transfer learning from read speech (POD-PT-FP) showed significantly lower pronunciation error rates compared to grapheme-level input or random initialization.
- Conversational Features: Training on fluent segments (without FPs) resulted in significantly fewer deleted function words and repeated syllables.
- Genre Appropriateness (MUSHRA Test):
- For read speech, the read-speech trained voice (READ-PR) was rated significantly higher.
- For public speaking and casual conversation, the podcast-trained voices (POD-PT-FP and POD-PT-FLU) were rated significantly higher than the read-speech voice.
- Authenticity: A pairwise preference test revealed that the voice trained with filled pauses (POD-PT-FP) was perceived as significantly more authentic than the fluent version (POD-PT-FLU).
Limitations and Future Work
- Limitations:
- The automated transcription, while effective, may still contain errors that impact synthesis quality.
- The breath detection method, while good, might miss some segments or include noise.
- The evaluation of FPs focused on their presence; their prosodic realization and placement are not deeply analyzed.
- Future Work:
- Exploring the impact of different FPs (uh vs. um) and their prosodic realization.
- Investigating the use of found data from other sources (e.g., interviews, lectures).
- Further refining automated transcription and segmentation for even cleaner data.
- Applying these findings to improve dialogue systems and other performative applications of synthetic speech.
Summary updated