Using Attention Networks and Adversarial Augmentation for Styrian Dialect Continuous Sleepiness and Baby Sound Recognition

Sung-Lin Yeh, Gao-Yi Chao, Bo-Hao Su, Yu-Lin Huang, Meng-Han Lin, Yin-Chun Tsai, Yu-Wen Tai, Zheng-Chi Lu, Chieh-Yu Chen, Tsung-Ming Tai, Chiu-Wang Tseng, Cheng-Kuang Lee, Chi-Chun Lee

Abstract

This paper presents a multi-faceted approach to the INTERSPEECH 2019 ComParE Challenge, focusing on Styrian dialect recognition, continuous sleepiness prediction, and baby sound classification. By integrating attention-based neural networks (CNN and BLSTM) with task-specific data augmentation techniques—including volume scaling, selective labeling of significant samples, and Adversarial Autoencoders (AAE)—the proposed systems consistently outperformed official baselines. Our results demonstrate the efficacy of attention mechanisms in capturing relevant acoustic cues and the necessity of tailored augmentation strategies to address data imbalance in computational paralinguistics.

Introduction

The field of computational paralinguistics aims to extract non-verbal information such as emotion, health status, or regional identity from speech. These tasks are inherently challenging due to the variability of real-world audio, including background noise and diverse recording conditions. A primary obstacle is data imbalance, where specific classes (e.g., rare infant vocalizations or regional dialects) are underrepresented. Additionally, subjective labeling in tasks like sleepiness estimation introduces ambiguity that can degrade model performance. This study explores how attention-based architectures and generative data augmentation can mitigate these issues by focusing on relevant acoustic cues and manufacturing high-quality training data.

Methods

The researchers developed a multi-pronged strategy utilizing two primary neural architectures and three distinct augmentation techniques tailored to each specific task.

Model Architectures

Two main architectures were employed to extract features from eGemaps Low-Level Descriptors (LLDs):

  1. Attention-based CNN: Acoustic features are treated as single-channel images. Convolutional layers extract spatial-temporal features, followed by a Bahdanau attention mechanism to weight the most informative frames before classification.
  2. Attention-based BLSTM: A Bidirectional Long Short-Term Memory network captures temporal dependencies, using an attention layer to summarize the sequence into a context vector.
Proposed model architectures for the three challenge tasks.

Task-Specific Augmentation and Processing

  • Styrian Dialect (SD): Robustness was improved through volume augmentation using scaling ratios of 0.5, 1, and 2. Interestingly, speed changes were found to be less effective for this specific task.
  • Continuous Sleepiness (CS): To reduce ambiguity, the model was trained on "Significant Labels"—the extreme ends of the 1–9 sleepiness scale (1–3 and 7–9)—filtering out the noisier middle-ground scores (4–6).
  • Baby Sound (BS): An Adversarial Autoencoder (AAE) was used to generate synthetic feature vectors for underrepresented classes like "laughing." External datasets were also used to prime the generative process for these sparse categories.

Model Fusion

Final predictions were generated through decision score fusion, averaging the outputs of the neural networks with traditional Support Vector Machines (SVMs) to ensure the final results were stable and robust across different data distributions.

Results

The proposed methods achieved superior performance across all three INTERSPEECH 2019 ComParE tasks compared to the official baselines.

Baby Sound Recognition

The system achieved an Unweighted Average Recall (UAR) of 62.39% on the test set, a significant 3.7% improvement over the baseline. The use of AAE-based augmentation was particularly effective for sparse classes like laughing and non-canonical babbling.

Performance comparison for the Baby Sound task, highlighting the gains from AAE-based augmentation.

Styrian Dialect and Continuous Sleepiness

For the Styrian Dialect task, the fusion model reached a UAR of 49.5%, representing a 2.5% absolute improvement. In the Continuous Sleepiness task, the system achieved a Spearman correlation of 0.369, surpassing the baseline by 0.026. These results confirm that focusing on extreme labels and utilizing attention mechanisms provides a clearer signal for the model to learn.

Conclusion

This work highlights the power of combining attention mechanisms with specialized data augmentation. Attention allows models to focus on discriminative acoustic segments, while generative models like AAEs provide the necessary data diversity to overcome sparsity in specialized datasets. Future research will explore more advanced generative models and domain-invariant feature learning to further improve performance in noisy, low-resource environments.

Summary updated

Abstract

In this study, we present extensive attention-based networks with data augmentation methods to participate in the INTERSPEECH 2019 ComPareE Challenge, specifically the three Sub-challenges: Styrian Dialect Recognition, Continuous Sleepiness Regression, and Baby Sound Classification. For Styrian Dialect Sub-challenge, these dialects are classified into Northern Styrian (NorthernS), Urban Sytrian (UrbanS), and Eastern Styrian (EasternS). Our proposed model achieves an UAR 49.5% on the test set, which is 2.5% higher than the baseline. For Continuous Sleepiness Sub-challenge, it is defined as a regression task with score range from 1 (extremely alert) to 9 (very sleepy). In this work, our proposed architecture achieves a Spearman correlation 0.369 on the test set, which surpasses the baseline model by 0.026. For Baby Sound Sub-challenge, the infant sounds are classified into canonical babbling, non-canonical babbling, crying, laughing and junk/other, and our proposed augmentation framework achieves an UAR of 62.39% on the test set, which outperforms the baseline by about 3.7%. Overall, our analyses demonstrate that by fusing attention network models with conventional support vector machine benefits the test set robustness, and the recognition rates of these paralinguistic attributes generally improve when performing data augmentation.
BibTeX
@inproceedings{yeh2019using,
  author = {Sung-Lin Yeh and Gao-Yi Chao and Bo-Hao Su and Yu-Lin Huang and Meng-Han Lin and Yin-Chun Tsai and Yu-Wen Tai and Zheng-Chi Lu and Chieh-Yu Chen and Tsung-Ming Tai and Chiu-Wang Tseng and Cheng-Kuang Lee and Chi-Chun Lee},
  title = {{Using Attention Networks and Adversarial Augmentation for Styrian Dialect Continuous Sleepiness and Baby Sound Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2110},
}
Figure