TL;DR
This paper investigates how both the predictable patterns of speech acoustics (bottom-up) and the predictability of words within a language (top-down) contribute to identifying sentence prominence. They found that unpredictability at both levels is a strong indicator of prominence, and their combined approach achieved high accuracy in detecting prominent words, even without explicit prominence labels during training.
What Problem Does This Paper Address?
Spoken language relies heavily on prosody – the rhythm, stress, and intonation of speech – to convey meaning beyond the literal words. A key aspect of prosody is "prominence," where certain words are emphasized to stand out. Understanding and automatically detecting prominence is crucial for applications like making text-to-speech systems sound more natural and improving speech recognition for tasks like content retrieval. While it's known that acoustic features (like pitch, energy, duration) and lexical predictability (how likely a word is to appear in a given context) both relate to prominence, their relative contributions and how they interact, especially in an unsupervised manner, were not fully understood. This paper aims to disentangle these contributions and explore their effectiveness in prominence detection systems.
Methods
The researchers explored two main approaches:
- Unsupervised Prominence Detection: This method aims to detect prominence without needing pre-labeled data. It models prominence as "surprisal" or unpredictability.
- Acoustic (Bottom-Up) Model: This model learns typical patterns of acoustic prosodic features (energy, fundamental frequency (F0), spectral tilt) over time. Deviations from these learned patterns are considered surprising and thus potentially prominent.
- Lexical (Top-Down) Model: This model uses language models (n-grams) to predict the likelihood of a word appearing in a given context. Words that are less predictable (lower probability) are considered more likely to be prominent.
- Combined Model: The scores from the acoustic and lexical models are combined to leverage both types of information.
- Supervised Classification: This approach uses standard machine learning classifiers (SVM and k-NN) trained on labeled data. They experimented with various combinations of acoustic features (mean, variance, etc., of F0, energy, tilt, duration) and lexical features (part-of-speech tags, n-gram probabilities) to see which ones perform best.
Datasets
The primary dataset used was the Spoken Dutch Corpus (Corpus Gesproken Nederlands; CGN). Specifically, they utilized the "Dutch news broadcast" section, which includes a subset with manually annotated prominence labels. This annotated subset contained approximately 44.3 minutes of speech from 10 speakers, with 7438 word tokens labeled for prominence. For training the language models in the unsupervised approach, a much larger corpus of newspaper and magazine articles (1.3 billion word tokens) was used.
Key Insights and Ideas
- Unpredictability is Key: The study strongly suggests that prominence is driven by unpredictability at multiple levels – both in the acoustic realization of speech and in the lexical predictability of words. This aligns with cognitive theories of attention and surprise.
- Lexical and Acoustic Cues are Complementary: Both bottom-up acoustic cues and top-down lexical cues are independently effective for prominence detection. Their combination, especially in the unsupervised system, leads to improved performance.
- Unsupervised Approach is Powerful: The unsupervised method, relying on predictability, achieved performance comparable to supervised methods, demonstrating its potential for practical applications where labeled data is scarce.
- Feature Importance Varies by Approach: While energy and F0 were strong acoustic cues in the unsupervised system, F0 and duration emerged as more informative acoustic features in the supervised setting. Lexical predictability (especially unigrams) was also highly effective in the unsupervised approach.
Key Results
- Unsupervised System: Achieved an accuracy of 87% using a combination of acoustic and lexical features. This is notably higher than previous unsupervised methods reported in the literature. The acoustic model alone reached 86% accuracy (using EN+FO), and the lexical model reached 82% (using unigrams).
- Supervised System: The best performing supervised combination (SVM with acoustic and lexical features) achieved an accuracy of 88%, showing high agreement with manual annotations. This performance is competitive with state-of-the-art approaches.
- Feature Correlation: A strong inverse correlation (r = -0.81) was found between n-gram probabilities and word durations, suggesting that common, predictable words tend to be shorter.
Limitations and Future Work
- Cross-Lingual Applicability: The authors acknowledge that direct comparison with other studies is difficult due to different corpora. They suggest that future work should investigate the contribution of these cues in different languages.
- Further Feature Exploration: While the study explored various combinations, there might be other acoustic or lexical features that could further improve performance.
- Refining Unsupervised Models: While the unsupervised model performed well, further research could explore more sophisticated ways to model acoustic and lexical predictability.
Summary updated