TL;DR
This paper explores how to better predict a speaker's certainty in spoken language by analyzing prosodic features within specific parts of an utterance, not just the whole thing. They found that combining features from the "target word" (the part likely causing uncertainty), its surrounding "context," and the "whole utterance" significantly improves prediction accuracy compared to using only whole-utterance features.
What Problem Does This Paper Address?
Understanding a speaker's certainty is crucial for many applications, like intelligent tutoring systems or language learning tools. While prosody (intonation, rhythm, stress) is known to convey emotional states and semantic meaning, precisely how sub-parts of an utterance contribute to signaling certainty has been less explored. Previous work often looked at prosodic features of entire utterances or intonational phrases. This paper aims to bridge this gap by investigating whether focusing on specific "salient" segments within an utterance, particularly those likely to be the source of uncertainty, can lead to more accurate certainty prediction.
Methods
The researchers developed a method to predict a speaker's perceived level of certainty using prosodic features. They collected a corpus of spoken utterances where a single word or phrase was identified as the likely source of the speaker's certainty or uncertainty. They then extracted a set of 20 prosodic feature types (including pitch, intensity, and temporal features) from three distinct segments: the whole utterance, the context (the fixed part of the response surrounding the target word), and the target word itself.
To build predictive models, they employed linear regression. A key innovation was the creation of a "combination" feature set. This set was constructed by selecting, for each prosodic feature type, the specific segment (whole utterance, context, or target word) that had the strongest correlation with the perceived level of certainty. This approach aimed to leverage the most informative prosodic cues without increasing the overall feature count. They used a 20-fold cross-validation strategy, with each fold testing on a different speaker to ensure generalization to unseen speakers.
Datasets
The study utilized a custom corpus of 600 utterances collected from 20 adult native English speakers. These utterances were elicited in two domains: public transportation in Boston and vocabulary completion tasks. For the purpose of this study, 480 utterances containing exactly one gap (i.e., one target word/phrase) were used. The utterances were annotated for certainty on a 5-point scale by five human judges, with an average inter-annotator agreement (Kappa) of 0.45.
Key Insights and Ideas
- Sub-utterance focus is key: The most significant insight is that analyzing prosodic features from specific segments (target word, context) alongside the whole utterance provides a richer signal for predicting certainty than relying solely on whole-utterance features.
- Target word as the locus of uncertainty: The "target word" or phrase, identified as the likely source of the speaker's decision-making, often contains strong prosodic cues related to their certainty.
- Context matters too: The surrounding "context" also contributes valuable prosodic information, suggesting that the way a phrase is embedded influences how certainty is perceived.
- Feature selection for efficiency: The "combination" feature set, which strategically selects the best prosodic feature from each segment type, achieves high accuracy with a reduced number of features (20) compared to using all features (60).
Key Results
The study found that:
- The combination feature set (E), with 20 features, achieved the highest prediction accuracies for both 5-class (45.42%) and 3-class (74.79%) certainty prediction using linear regression.
- This combination set outperformed the whole utterance feature set (A) by a significant margin (6.42% for 5 classes, 5.83% for 3 classes).
- The "All features" set (D), with 60 features, also performed well (48.54% for 5 classes, 74.58% for 3 classes), but the combination set achieved comparable or slightly better results with fewer features.
- Cross-validation analysis showed that the combination set's predictions were more strongly correlated with perceived certainty in 16 out of 20 folds compared to the utterance-only set.
Limitations and Future Work
- Comparison to prior work: While the best accuracy (74.79% for 3 classes) was slightly lower than a previous study (Liscombe et al., 2005, 76.42%), the authors note that their study used a more rigorous cross-validation approach and a larger difference from the naive baseline, suggesting their results might be more robust. The exact testing methodology of the prior work is unclear.
- Speaker-dependent normalization: Pitch and intensity features were normalized by speaker, which is a standard practice but might limit direct comparison across speakers without further analysis.
- Future Work: The authors plan to experiment with predicting the speaker's self-reported level of certainty, which could offer a different perspective and potentially validate their findings further.
Summary updated