TL;DR
This paper introduces a novel approach to paralinguistic classification by leveraging the posterior outputs of Acoustic Speech Recognition (ASR) Deep Neural Networks (DNNs) within a Bag-of-Audio-Words (BoAW) framework. By training ASR models on large, general speech corpora and then applying BoAW to their frame-level phonetic posteriors, the authors achieve state-of-the-art results on a challenging paralinguistic dataset, demonstrating the power of combining large-scale ASR data with paralinguistic tasks.
What Problem Does This Paper Address?
Computational paralinguistics aims to extract and identify non-verbal aspects of speech, such as emotion, conflict intensity, or even medical conditions. A key challenge in this field is the need to represent variable-length utterances as fixed-length feature vectors for machine learning models. Traditional methods often rely on low-level acoustic features (like MFCCs) and statistical functionals (mean, standard deviation) computed over utterance frames. While effective, these methods can be limited by the size and diversity of available paralinguistic datasets, which are often much smaller than those used for ASR. The paper addresses the gap of how to effectively leverage the vast amounts of data available for ASR training to improve paralinguistic classification, especially when paralinguistic datasets are limited.
Methods
The core innovation lies in using the output of a pre-trained ASR DNN as input for the BoAW feature extraction process. Here's a breakdown:
- ASR DNN Training: A context-independent (CI) DNN acoustic model is trained on a large, general speech corpus (the "external" corpus). This DNN is designed for phoneme identification.
- ASR DNN Evaluation: The trained DNN is then evaluated on the frame-level features of the target paralinguistic dataset. Instead of using the raw acoustic features, the paper uses the posterior probabilities output by the DNN for each phoneme state at each frame. These posteriors represent the DNN's confidence in different phonetic interpretations.
- BoAW Feature Extraction: The BoAW framework is applied to these frame-level DNN posterior vectors. This involves:
- Clustering the posterior vectors to create a "codebook."
- Replacing each frame's posterior vector with the index of its nearest codeword.
- Constructing an utterance-level feature vector as a normalized histogram of these codeword indices.
- Feature Combination (Late Fusion): The BoAW features derived from DNN posteriors are combined with other feature sets (standard ComParE features and BoAW features derived from MFCCs) using late fusion. This involves training separate classifiers for each feature set and then combining their posterior predictions, typically through a weighted average.
This approach differs from standard BoAW by using DNN posteriors instead of raw acoustic features and by leveraging an external, large-scale ASR corpus for training the DNN, which is then applied to the paralinguistic task.
Datasets
- External Corpus (for ASR DNN training): "Szeged" Hungarian broadcast news corpus (28 hours), specifically a 22-hour training subset. This corpus is used to train the ASR DNN.
- Paralinguistic Dataset (for evaluation): The iHEARu-EAT corpus [27]. This dataset contains 2 hours and 51 minutes of German speech from 30 speakers recorded while eating six types of food, plus a "no food" class, resulting in 7 classes. It was previously used in the Interspeech ComParE 2015 Eating Condition Sub-Challenge.
Key Insights and Ideas
- Leveraging ASR Data for Paralinguistics: The paper demonstrates that large, general ASR datasets, even if in a different language (Hungarian vs. German), can significantly benefit paralinguistic tasks when their DNN models' outputs are used as features. This is a powerful way to overcome the data scarcity often faced in specialized paralinguistic domains.
- DNN Posteriors as Rich Features: The posterior probabilities from an ASR DNN capture nuanced phonetic information that goes beyond raw acoustic features. These posteriors, when clustered and histogrammed via BoAW, provide a robust utterance-level representation.
- BoAW's Adaptability: The BoAW framework proves flexible enough to work with various frame-level representations, including DNN posteriors, and can be effectively combined with traditional feature sets.
- Importance of Feature Combination: The best performance is achieved not by a single feature set but by combining the BoAW-DNN posteriors with standard ComParE features and BoAW-MFCC features, highlighting the complementary nature of different feature extraction strategies.
Key Results
- BoAW-DNN Alone: The BoAW representation derived from ASR DNN posteriors, when used alone, achieved competitive scores, though it lagged behind standard paralinguistic approaches and required large codebook sizes.
- Combined Approach: The proposed approach, combining BoAW-DNN posteriors with standard ComParE features and BoAW-MFCC features via late fusion, achieved the highest Unweighted Average Recall (UAR) score reported to date on the iHEARu-EAT dataset in a scientifically sound manner.
- Relative Error Reduction: The combination of BoAW-DNN with ComParE features yielded a 10% improvement in relative error reduction. Further combining with BoAW-MFCC features resulted in a significant 32% relative error reduction compared to the baseline.
- Optimal Codebook Size: For both BoAW-MFCC and BoAW-DNN approaches, larger codebook sizes (up to 16384) led to better performance, indicating that finer granularity in the "audio word" vocabulary improves classification.
Limitations and Future Work
- Codebook Size: The need for very large codebook sizes to achieve optimal performance leads to large feature vectors, posing computational challenges.
- Language Mismatch: While the approach worked despite a language mismatch between the training (Hungarian) and evaluation (German) datasets, the impact of language and external dataset size on BoAW quality warrants further investigation.
- Paralinguistic Task Dependency: The effectiveness of BoAW-DNN posteriors might vary across different paralinguistic tasks (e.g., emotion vs. stress detection).
- DNN Output Interpretation: The authors suggest exploring the DNN output vectors as samples from a probability distribution for potential improvements in codebook construction.
- Further Investigations: Future work could explore different distance functions for clustering, investigate the impact of external corpus size and language, and test the approach on a wider range of paralinguistic tasks.
Summary updated