TL;DR
This paper investigates the intricate relationship between pauses, breathing patterns, and linguistic structure in book reading. By analyzing a large corpus of audiobook recordings, the authors demonstrate that respiratory signals, particularly inhalation timing, can serve as valuable cues for understanding discourse structure, complementing traditional prosodic markers. This work has implications for more natural and human-like text-to-speech synthesis.
What Problem Does This Paper Address?
Creating natural-sounding text-to-speech (TTS) systems requires more than just accurate pronunciation; it demands a deep understanding of how humans structure spoken language. This includes not only linguistic and prosodic elements but also the often-overlooked role of breathing. Pauses in speech are not random silences but are strategically placed to convey meaning and manage information flow. Furthermore, breathing patterns are intrinsically linked to speech production, serving to refill the lungs and pace discourse. Existing TTS models often struggle to adequately capture these complex interactions, leading to robotic or unnatural speech. This paper aims to bridge this gap by exploring how pauses and respiratory markers can be analyzed and potentially modeled to improve speech synthesis.
Methods
The authors analyze a corpus of book readings, focusing on pauses and the acoustic markers of inhalation. They categorize pauses into four types: syntactic (short, no inhalation), sentence-internal (mid-to-long, with or without inhalation, often at commas or major syntactic boundaries), sentence-final (at full stops), and end-of-paragraph.
For pause prediction, they employ two main approaches:
- Syntax-based prediction: Using the Wapiti tool and decision trees (C4.5) with Part-of-Speech (POS) tags of surrounding words and punctuation as input.
- Phonotactic constraints: Integrating a Markov Decision Process (MDP) that links states representing patterns of successive words and pauses, incorporating transition and emission probabilities derived from pause distributions.
To analyze breathing, they identify inhalation noises in spectrograms, noting their presence and timing relative to pauses. They also compute "Inter P-Center Group" (IPCG) durations, which include the preceding rime and following onset, to better capture final lengthening effects.
Datasets
The primary dataset consists of the first 15 chapters of the French version of Jules Verne's "Around the World in 80 Days," read by a single speaker. The audio data (2 hours 35 minutes) was automatically aligned with a phonetic transcription tagged with POS information, which was then manually corrected. This corpus represents a significant collection of natural book reading, providing a rich source for analyzing pause and respiration patterns.
Key Insights and Ideas
- Breathing as a Discourse Cue: Inhalation timing, not just pause duration, can signal thematic continuity or discontinuity. Delayed breaths, especially with rising tones, can indicate upcoming complementary information, while early or absent breaths with falling tones might signal the end of a theme.
- Quantal Effects in Pausing: Pause durations exhibit multimodal distributions that are well-explained by underlying lognormal modes. These modes appear to correspond to "quantal" effects, suggesting that pause durations are often close to integer multiples of a "mandible cycle," hinting at a fundamental rhythmic constraint.
- Syntactic Structure Dominates: While phonotactic information plays a role, syntactic structure (POS tags and punctuation) is a strong predictor of pause locations, with pauses frequently occurring at major syntactic breaks.
- Inhalation is Common: A significant majority (69.7%) of mid- and long-duration pauses include audible breath sounds, highlighting the integral role of respiration in speech production.
- Speaker-Specific Rhythms: The observed rhythmic patterns and pause durations might be influenced by speaker and language, suggesting that personalized modeling could be beneficial for TTS.
Key Results
- Pause Location Prediction: A syntax-based approach achieved an accuracy of close to 0.95 and a F-score of 0.8. Removing punctuation reduced accuracy to 0.93.
- Pause Duration Prediction: Regression trees yielded an overall correlation of 0.72 with original pause durations. Incorporating phonotactic information slightly increased this correlation to 0.76, with sentence-final pauses showing a stronger influence from preceding phrase length.
- Inhalation Presence: 69.7% of mid- and long-duration pauses contained audible breath sounds.
- Average Phonation Rate: 5.67 syllables/s.
Limitations and Future Work
- Single Speaker: The analysis is based on a single speaker, which may limit the generalizability of the findings. Future work should investigate a wider range of speakers and styles.
- Subjective Labeling: Inhalation loci were detected based on listening, which is inherently subjective. Synchronous recordings of physiological signals (e.g., respiratory movements) are planned to provide more objective data.
- Perception Tests: The authors plan to conduct perception tests to understand the subjective impact of different breathing and pausing strategies on listener comprehension and naturalness.
- TTS Integration: While the findings provide valuable insights for TTS, direct integration and evaluation of these models in a synthesis system are not discussed.
- Language Specificity: The observed "quantal" effects might be specific to French or the reading style, warranting further investigation across different languages.
Summary updated