TL;DR
This paper investigates the prosody of Swedish conversational grunts (like "mhm" and "eh") to understand how they signal attitude and turn-taking intentions. By using a novel supra-segmental prosodic representation and contextual features, they show that these grunts carry significant information, allowing for classification of their meaning and predicting conversational flow.
What Problem Does This Paper Address?
Conversational speech is rich with non-lexical sounds like fillers ("um") and feedback tokens ("mhm," "yes") that are crucial for natural human interaction. Automatic dialog systems struggle to interpret these sounds, which can convey subtle meanings about a speaker's attitude (e.g., agreement, disagreement) and their intention to take the conversational turn. This paper tackles the challenge of automatically recognizing these meanings by analyzing the prosodic features of conversational grunts, specifically in Swedish. The authors aim to bridge the gap between human intuition about these sounds and computational models that can process them effectively.
Methods
The core of this work lies in a novel supra-segmental prosodic signal representation based on Time Varying Constant-Q Cepstral Coefficients (TVCQCC). This method is designed to capture varying lengths of speech segments and is particularly suited for modeling the fundamental frequency (F0) region of spectra without complex pitch tracking. They also incorporate contextual timing features and the "end of interlocutor left context" to improve classification. For classification, Linear Discriminant Analysis (LDA) is employed, using a diagonal covariance matrix for robustness. The TVCQCC matrices are converted into fixed-size vectors by stacking rows, and then mean F0 and token length are added as features.
Datasets
The study utilizes data from the DEAL corpus. This corpus consists of informal, human-human, face-to-face conversations recorded with 6 subjects (4 male, 2 female) acting as shopkeepers and buyers in a flea market scenario. The dialogs are transcribed orthographically, including non-lexical entities, and are rich in fillers and feedback tokens. Feedback tokens are manually annotated for perceived attitudes: news receiving, dis-preference, or general feedback. Additionally, all tokens are annotated for their turn-taking effect (Same Speaker, Other Speaker, or Simultaneous Start).
Key Insights and Ideas
- Grunts are information-rich: Conversational grunts, often overlooked, carry significant prosodic information that signals both speaker attitude and turn-taking intent.
- Prosody matters for attitude: The way a grunt is spoken (its pitch, duration, and spectral characteristics) can differentiate between attitudes like "news receiving" and "dis-preference."
- Turn-taking prediction is possible: The prosodic and contextual features of grunts can predict whether the speaker intends to continue speaking, yield the floor, or if a simultaneous start is likely.
- Context is crucial for turn-taking: The "left context" of the interlocutor's speech significantly boosts the performance of turn-taking prediction, especially for distinguishing between yielding the floor and simultaneous starts.
- Turn-initial feedback loses some nuance: Grunts that initiate a turn (turn-initial feedback) show fewer attitude-signaling prosodic cues compared to grunts that occur mid-turn as continuers. This suggests that the immediate turn-taking goal might overshadow nuanced attitudinal expression.
Key Results
- Filler vs. Feedback Distinction: The proposed supra-segmental representation and features achieve a recall rate of 79.8% (without context) and 82.1% (with context) for distinguishing between fillers and feedback tokens (Task 1).
- Attitude Classification: Classification of feedback attitudes (news receiving, dis-preference, general feedback) is possible above chance level, with better performance when the feedback token yields the floor (Task 2b and 2c show higher recall than Task 2a).
- Turn-Taking Prediction: The turn-taking effect of feedbacks can be predicted with above chance recall (e.g., 63.9% without context, 66.0% with context for Task 3b, where Simultaneous Starts are removed).
- Simultaneous Starts are tricky: Predicting simultaneous starts is challenging, with recall rates below chance. However, merging them with "Other Speaker" conditions yields better results than merging them with "Same Speaker," suggesting they are more akin to floor-stealing attempts.
- Timing Features Boost Performance: Adding timing features consistently improves recall across most tasks.
- Left Context Boosts Turn-Taking: The interlocutor's left context significantly improves performance for turn-taking prediction tasks (Tasks 3a-d).
Limitations and Future Work
- Simultaneous Starts: The paper acknowledges the difficulty in predicting "Simultaneous Starts" and suggests that further investigation into the specific cues leading to these events is warranted.
- Filler Turn-Taking: The turn-taking effect of fillers was not addressed due to insufficient training data, indicating a potential area for future research.
- Attitude Nuance: While classification of attitudes is possible, the paper notes that turn-initial feedback tokens lose some of their attitude-signaling prosodic cues. Future work could explore how these nuances are conveyed in other ways or if more sophisticated features can recover them.
- Generalizability: The study focuses on Swedish conversational grunts. Future work could explore the cross-linguistic applicability of these findings and methods.
Summary updated