Abstract
This study quantifies the redundancy between prosody—the rhythm and melody of speech—and linguistic context using information theory and Large Language Models (LLMs). By analyzing the LibriTTS corpus, researchers identified a fundamental asymmetry: a word's prosody is predictable from a relatively long history of past words (up to 8), but only from a very narrow window of future words (1–2). These findings reflect the constraints of human memory and incremental speech planning, suggesting that prosody serves as a cognitive buffer for integrating past information while signaling immediate future intent.
Introduction
When we speak, we do more than just transmit words; we employ pitch, loudness, and timing to signal meaning, emphasis, and structure. While previous research has established that prosody is largely "redundant"—meaning it can often be predicted from the text alone—the specific dynamics of this relationship remained unclear. How much context does a listener or an AI actually need to predict how a word will be uttered? Does a speaker's prosody reflect a plan for the next ten words, or just the next one? This paper addresses these questions to better understand the time scales of human language processing and the communicative roles of prosody.
Methodology
The researchers framed the problem using Mutual Information (MI), a metric from information theory that quantifies how much knowing one variable (linguistic context) reduces uncertainty about another (prosodic features).
Estimation via LLMs
Since the true probability distributions of speech are unknown, the authors used fine-tuned Large Language Models (specifically BERT-large and RoBERTa-large) to predict prosodic features from text. The "loss" of these models serves as a proxy for conditional entropy. They systematically varied the amount of context provided to the model, ranging from 0 to 9 words in both the past ($n$) and future ($m$) directions.
Data and Features
The study utilized LibriTTS, a large-scale corpus of English audiobooks containing approximately 585 hours of speech. The Montreal Forced Aligner (MFA) was used to synchronize text and audio, followed by the extraction of six distinct prosodic features: pitch ($f_0$), loudness (energy), duration, pauses, and two measures of prominence (absolute and relative).
Results
The analysis revealed a stark difference between how past and future context relate to prosody, supporting theories of incremental planning and memory constraints.
The Asymmetry of Speech
Prosody appears to "look back" much further than it "looks ahead." For almost all features, MI with past context was significantly higher than with future context at longer scales. While redundancy with past context plateaus at about 5–8 words, redundancy with future context plateaus almost immediately, at 1–2 words.
Feature-Specific Trends
- Pitch, Loudness, and Prominence: These features followed the general trend of long-past/short-future redundancy.
- Duration and Pause: These were notable exceptions. They showed very weak redundancy with the past but relatively high redundancy with the future. This suggests that slowing down or pausing primarily serves to prepare for the upcoming word or to signal a structural boundary.
Discussion
The findings suggest that prosody facilitates communication by helping listeners anchor the current word in the established context (long-term) while simultaneously predicting upcoming material (short-term). The short-scale future planning supports the theory of "incremental planning," where speakers only have a concrete plan for the immediate next word or two.
Limitations and Future Directions
The study's reliance on "read" speech (audiobooks) may not fully capture the dynamics of spontaneous conversation. Furthermore, as the study was limited to English, future work should explore whether these redundancy time scales vary in tonal or pitch-accent languages. The researchers also suggest investigating whether these "redundancy time scales" align with the "temporal receptive windows" observed in human brain activity during language processing.
Summary updated