TL;DR
This paper tackles the challenge of automatically distinguishing between spoken words and non-linguistic vocalizations (like laughter or sighs) in spontaneous speech. They demonstrate that while models trained on one dataset perform poorly on another, combining data from multiple sources significantly improves generalization, suggesting a path towards more robust speech understanding systems.
What Problem Does This Paper Address?
In human conversation, understanding not just what is said (linguistic content) but also how it's said (paralinguistic cues) is crucial for natural interaction. Non-linguistic vocalizations, such as laughter, sighs, hesitations ("um," "uh"), and vocal noises, carry significant emotional and intentional information. However, automatically recognizing and distinguishing these from actual spoken words in spontaneous, real-world speech is a difficult problem. Existing research often focuses on detecting a single type of vocalization or evaluates models only within the dataset they were trained on (intra-corpus). This paper addresses the gap of evaluating how well models generalize to unseen data from different sources (inter-corpus) and explores strategies to improve this generalization.
Methods
The authors employ a standard approach using Hidden Markov Models (HMMs) for classification. They extract Mel frequency cepstral coefficients (MFCCs) along with their derivatives as acoustic features. A key aspect of their methodology is the experimental setup:
- Five-Class Task: They define a classification task with five categories: words, laughter, vocal noise (including breathing, sighing, coughing), non-verbal consent ("mhm"), and filled pauses ("um," "uh").
- Speaker-Independent Training/Testing: All experiments are conducted in a speaker-independent manner to ensure models generalize beyond specific individuals.
- Intra-Corpus Evaluation: Models are trained and tested on data from the same corpus.
- Inter-Corpus Evaluation: This is the core of their contribution. They explore two scenarios:
- Pairwise: Training on one corpus and testing on another.
- Leave-One-Corpus-Out (LOCO): Training on the union of all corpora except one, and testing on the held-out corpus. This simulates a scenario where you have a large, diverse training set and want to test on a new, unseen dataset.
- Data Agglomeration: They investigate whether combining data from multiple corpora improves inter-corpus performance.
Datasets
The study utilizes four publicly available spontaneous English speech corpora:
- AVEC (Audio/Visual Emotion Challenge): Part of the SEMAINE corpus, featuring human-operator conversations designed to elicit emotional responses.
- TUM AVIC: Used in the INTERSPEECH 2010 Paralinguistic Challenge, involving product presentations with subject interaction.
- Buckeye Corpus: A large corpus of natural, spontaneous interviews with native speakers.
- COSINE Corpus: Features multi-party conversations recorded in real-world, noisy environments.
These corpora vary in size, recording conditions, and conversational scenarios, providing a diverse testbed for inter-corpus generalization.
Key Insights and Ideas
- Inter-corpus performance is significantly lower than intra-corpus: This is a fundamental finding, highlighting the challenge of applying models trained on one dataset to another. Acoustic conditions, speaker variability, and domain differences all contribute to this drop.
- Data agglomeration is key to generalization: Combining data from multiple diverse corpora (LOCO strategy) dramatically improves performance on unseen test sets compared to training on a single corpus. This suggests that "more data" from varied sources is indeed beneficial.
- Corpus "prototypicality" matters: The Buckeye corpus consistently shows the best performance, both intra- and inter-corpus. This suggests that its characteristics (naturalness, formality dissipation) make it a good source for building robust, generalizable models, not just its size.
- Accuracy vs. UAR trade-off: While accuracy often increases with more model complexity (e.g., more Gaussian mixtures), the Unweighted Average Recall (UAR) – which better reflects performance on minority classes – can decrease. This is an important consideration for imbalanced datasets.
- Consent class is challenging: The "consent" class ("mhm") is often underrepresented, particularly in the AVEC corpus, leading to very low recall for this class.
Key Results
- Intra-corpus UAR: The Buckeye corpus achieved the highest average UAR (82.9%), while AVEC had the lowest (67.1%), attributed to data sparsity.
- Pairwise Inter-corpus UAR: Generally much lower than intra-corpus results. For example, training on Buckeye and testing on COSINE yielded a UAR of 62.0%, while intra-corpus Buckeye UAR was around 82.9%. A notable incompatibility was observed between AVIC and AVEC, with UAR dropping below 30%.
- LOCO Inter-corpus UAR: Significantly improved performance over pairwise evaluation. For instance, testing on AVEC using LOCO yielded a UAR of 61.4%, compared to 49.6% in pairwise testing.
- Data Agglomeration Gains: LOCO evaluation consistently showed significant gains over single-corpus training for all test corpora (p < 0.001).
Limitations and Future Work
- Pre-segmented Data: The study focuses on classifying pre-segmented vocalizations, not detecting them in continuous speech, which is acknowledged as a more challenging task.
- Coarse Class Definition: A five-class task was chosen for unification across corpora, meaning some finer-grained distinctions (e.g., different types of vocal noise) were not explored.
- Future Directions: The authors suggest exploring semi-supervised learning, using unlabeled spontaneous speech from the internet, and investigating cross-cultural effects in vocalization classification. They also propose unsupervised adaptation of agglomerated datasets.
Summary updated