Abstract
This paper presents a multi-faceted approach to the INTERSPEECH 2019 ComParE Challenge, focusing on Styrian dialect recognition, continuous sleepiness prediction, and baby sound classification. By integrating attention-based neural networks (CNN and BLSTM) with task-specific data augmentation techniques—including volume scaling, selective labeling of significant samples, and Adversarial Autoencoders (AAE)—the proposed systems consistently outperformed official baselines. Our results demonstrate the efficacy of attention mechanisms in capturing relevant acoustic cues and the necessity of tailored augmentation strategies to address data imbalance in computational paralinguistics.
Introduction
The field of computational paralinguistics aims to extract non-verbal information such as emotion, health status, or regional identity from speech. These tasks are inherently challenging due to the variability of real-world audio, including background noise and diverse recording conditions. A primary obstacle is data imbalance, where specific classes (e.g., rare infant vocalizations or regional dialects) are underrepresented. Additionally, subjective labeling in tasks like sleepiness estimation introduces ambiguity that can degrade model performance. This study explores how attention-based architectures and generative data augmentation can mitigate these issues by focusing on relevant acoustic cues and manufacturing high-quality training data.
Methods
The researchers developed a multi-pronged strategy utilizing two primary neural architectures and three distinct augmentation techniques tailored to each specific task.
Model Architectures
Two main architectures were employed to extract features from eGemaps Low-Level Descriptors (LLDs):
- Attention-based CNN: Acoustic features are treated as single-channel images. Convolutional layers extract spatial-temporal features, followed by a Bahdanau attention mechanism to weight the most informative frames before classification.
- Attention-based BLSTM: A Bidirectional Long Short-Term Memory network captures temporal dependencies, using an attention layer to summarize the sequence into a context vector.
Task-Specific Augmentation and Processing
- Styrian Dialect (SD): Robustness was improved through volume augmentation using scaling ratios of 0.5, 1, and 2. Interestingly, speed changes were found to be less effective for this specific task.
- Continuous Sleepiness (CS): To reduce ambiguity, the model was trained on "Significant Labels"—the extreme ends of the 1–9 sleepiness scale (1–3 and 7–9)—filtering out the noisier middle-ground scores (4–6).
- Baby Sound (BS): An Adversarial Autoencoder (AAE) was used to generate synthetic feature vectors for underrepresented classes like "laughing." External datasets were also used to prime the generative process for these sparse categories.
Model Fusion
Final predictions were generated through decision score fusion, averaging the outputs of the neural networks with traditional Support Vector Machines (SVMs) to ensure the final results were stable and robust across different data distributions.
Results
The proposed methods achieved superior performance across all three INTERSPEECH 2019 ComParE tasks compared to the official baselines.
Baby Sound Recognition
The system achieved an Unweighted Average Recall (UAR) of 62.39% on the test set, a significant 3.7% improvement over the baseline. The use of AAE-based augmentation was particularly effective for sparse classes like laughing and non-canonical babbling.
Styrian Dialect and Continuous Sleepiness
For the Styrian Dialect task, the fusion model reached a UAR of 49.5%, representing a 2.5% absolute improvement. In the Continuous Sleepiness task, the system achieved a Spearman correlation of 0.369, surpassing the baseline by 0.026. These results confirm that focusing on extreme labels and utilizing attention mechanisms provides a clearer signal for the model to learn.
Conclusion
This work highlights the power of combining attention mechanisms with specialized data augmentation. Attention allows models to focus on discriminative acoustic segments, while generative models like AAEs provide the necessary data diversity to overcome sparsity in specialized datasets. Future research will explore more advanced generative models and domain-invariant feature learning to further improve performance in noisy, low-resource environments.
Summary updated