The INTERSPEECH 2019 Computational Paralinguistics Challenge: Styrian Dialects, Continuous Sleepiness, Baby Sounds & Orca Activity

Björn W. Schuller, Anton Batliner, Christian Bergler, Florian B. Pokorny, Jarek Krajewski, Margaret Cychosz, Ralf Vollmann, Sonja-Dana Roelen, Sebastian Schnieder, Elika Bergelson, Alejandrina Cristia, Amanda Seidl, Anne S. Warlaumont, Lisa Yankowitz, Elmar Nöth, Shahin Amiriparian, Simone Hantke, Maximilian SchmittInterspeech 2019

TL;DR

The INTERSPEECH 2019 Computational Paralinguistics Challenge introduced four novel sub-challenges: Styrian dialect classification, continuous sleepiness assessment, baby sound classification, and orca sound detection. This paper details the challenge setup, baseline methods (including traditional features and deep learning), and results, aiming to advance the state-of-the-art in these diverse paralinguistic tasks.

What Problem Does This Paper Address?

Computational paralinguistics, the study of human expressive and affective signals in speech, has seen significant progress. However, existing research often focuses on a limited set of well-defined tasks. This paper addresses the need to expand the scope of computational paralinguistics research by introducing and standardizing evaluation for four new, practically relevant, yet challenging tasks:

  • Styrian Dialects: Differentiating between subtle regional variations of Austrian-German dialects is crucial for applications requiring fine-grained linguistic understanding, such as localized voice assistants or forensic analysis.
  • Continuous Sleepiness: Accurately assessing a speaker's level of sleepiness in real-time is vital for safety-critical applications like driver assistance systems, where drowsiness can have severe consequences.
  • Baby Sounds: Classifying different types of infant vocalizations (e.g., babbling, crying, laughing) can aid in early diagnostics for developmental delays and understanding infant communication.
  • Orca Activity: Detecting orca sounds in underwater recordings is essential for bioacoustic research, enabling better monitoring of marine ecosystems and the behavior of these vocal marine mammals.

The challenge aims to provide a common ground for researchers to develop and compare methods for these diverse tasks.

Methods

The challenge provided participants with baseline feature extraction and classification methods, while also allowing for novel approaches. Key methods described include:

  • ComParE Acoustic Feature Set: A standard set of acoustic features, including functionals (statistics) computed over low-level descriptors (LLDs), used in previous INTERSPEECH challenges.
  • Bag-of-Audio-Words (BoAW): This approach represents audio chunks as histograms of acoustic LLDs, quantized using a codebook. Separate codebooks are learned for LLDs and their deltas, and the resulting representations are concatenated.
  • AUDEEP (Unsupervised Representation Learning): This method utilizes deep learning, specifically sequence-to-sequence autoencoders with Recurrent Neural Networks (RNNs), to learn representations from audio without explicit labels. Mel-scale spectrograms are processed, and learned feature vectors are concatenated.
  • Support Vector Machines (SVM) and Support Vector Regression (SVR): Linear SVMs were used for classification tasks, and SVR for the regression task (sleepiness). These were implemented using the Scikit-learn toolkit.
  • Data Preprocessing: All audio was converted to a single-channel, 16 kHz, 16-bit PCM format, with specific handling for the Orca Activity sub-challenge. Feature representations were scaled to zero mean and unit standard deviation.

Datasets

The challenge utilized several distinct datasets for its four sub-challenges:

  • Styrian Dialects (STYRIALECTS): Recordings of Austrian-German speakers from different regions of Styria, automatically segmented and manually revised into utterances of varying lengths. Three dialects were targeted: Northern Styrian, Urban Styrian, and Eastern Styrian.
  • Continuous Sleepiness (SLEEP Corpus): A subset of the Düsseldorf Sleepy Language Corpus, containing recordings of subjects reporting their sleepiness using the Karolinska Sleepiness Scale (KSS). Both self-assessments and post-hoc observer ratings were used as reference.
  • Baby Sounds (Baby Sounds Sub-Challenge): A dataset comprising 12,445 vocalizations from 46 healthy infants (2-36 months old) exposed to various languages. Vocalizations were segmented into chunks and categorized into five types: canonical babbling, non-canonical babbling, crying, laughing, and junk/other.
  • Orca Activity (DeepAL Fieldwork Data - DLFD): Underwater acoustic recordings collected in Northern British Columbia. This dataset was used to detect orca sounds versus general noise.

Key Insights and Ideas

  • Diversity of Paralinguistic Signals: The challenge highlights the broad spectrum of paralinguistic signals, ranging from subtle dialectal variations to distinct emotional and biological sounds.
  • Value of Unsupervised Learning: The inclusion of AUDEEP demonstrates the growing potential of deep unsupervised learning methods for extracting meaningful features from audio, reducing reliance on manual feature engineering.
  • Data Sparsity and Imbalance: Several sub-challenges faced issues with data sparsity (e.g., extreme sleepiness levels) or class imbalance (e.g., laughing in baby sounds), posing significant challenges for robust modeling.
  • Importance of Evaluation Metrics: The use of Unweighted Average Recall (UAR) and Spearman's Correlation Coefficient (ρ) underscores the need for appropriate metrics that account for class imbalance and the nature of regression tasks.
  • Reproducibility: Providing baseline scripts and detailed descriptions aims to foster reproducibility and allow participants to build upon existing work.

Key Results

The paper presents baseline results for each sub-challenge, achieved using the described methods:

  • Styrian Dialects: Baseline UAR of 47.0%.
  • Continuous Sleepiness: Baseline Spearman's ρ of .343.
  • Baby Sounds: Baseline UAR of 58.7%.
  • Orca Activity: Baseline AUC of .866.

Confusion matrices are provided for the development sets, illustrating the performance and common misclassifications for each task. For instance, the Baby Sounds confusion matrix shows good performance on the diagonal, indicating successful classification of most instances, while other tasks exhibit more significant off-diagonal confusion.

Limitations and Future Work

  • Limited End-to-End Learning: The AUDEEP approach, while using deep learning, did not explore end-to-end learning directly from raw audio signals.
  • Hyperparameter Optimization: For AUDEEP, hyperparameters were not fully optimized for the presented baseline results.
  • Test Set Blinding: Test set distributions were blinded during the challenge, with results presented here being from the development set for baselines.
  • Future Work: The authors expect participants to achieve better performance by employing novel procedures and features tailored to the specific tasks. They also emphasize the ongoing relevance of these paralinguistic tasks for real-world applications. The challenge itself serves as a catalyst for future research in these areas.

Summary updated

BibTeX
@inproceedings{schuller2019interspeech,
  author = {Björn W. Schuller and Anton Batliner and Christian Bergler and Florian B. Pokorny and Jarek Krajewski and Margaret Cychosz and Ralf Vollmann and Sonja-Dana Roelen and Sebastian Schnieder and Elika Bergelson and Alejandrina Cristia and Amanda Seidl and Anne S. Warlaumont and Lisa Yankowitz and Elmar Nöth and Shahin Amiriparian and Simone Hantke and Maximilian Schmitt},
  title = {{The INTERSPEECH 2019 Computational Paralinguistics Challenge: Styrian Dialects, Continuous Sleepiness, Baby Sounds \& Orca Activity}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure