Using Fisher Vector and Bag-of-Audio-Words Representations to Identify Styrian Dialects, Sleepiness, Baby & Orca Sounds

Gábor GosztolyaInterspeech 2019

TL;DR

This paper explores the effectiveness of Fisher Vectors (FV) and Bag-of-Audio-Words (BoAW) as general-purpose feature representations for paralinguistic tasks, aiming to outperform traditional methods. The authors found that FV and BoAW, particularly FV, offered significant improvements in three out of four ComParE 2019 challenge sub-tasks, demonstrating their robustness and potential for diverse audio analysis applications.

What Problem Does This Paper Address?

Human speech carries a wealth of information beyond the spoken words, encompassing the speaker's emotional state, physical condition, and even the surrounding environment. The INTERSPEECH Computational Paralinguistics Challenge (ComParE) aims to develop automatic systems for identifying these "paralinguistic" cues. While many approaches focus on task-specific features, there's a growing interest in developing general-purpose feature representations that can be applied across various tasks with minimal modification. This paper investigates the utility of Fisher Vectors and Bag-of-Audio-Words as such general-purpose representations, comparing them against the established "ComParE functionals" on four distinct paralinguistic tasks: dialect identification, sleepiness estimation, baby sound categorization, and orca sound detection.

Methods

The authors experimented with three primary feature representation techniques:

  • ComParE functionals: This is the established baseline feature set, comprising 6373 utterance-level features derived from frame-level acoustic attributes (energy, spectral, MFCCs, voicing).
  • Fisher Vectors (FV): Adapted from image processing, FV models the distribution of low-level features (here, MFCCs) using Gaussian Mixture Models (GMMs). The Fisher score, which is the gradient of the log-likelihood function with respect to the GMM parameters, captures how much the data deviates from the model. This results in a fixed-length utterance-level representation. The authors experimented with different numbers of Gaussian components and variations of MFCCs (e.g., MFCC+Δ, MFCC+Δ+ΔΔ).
  • Bag-of-Audio-Words (BoAW): This method involves clustering frame-level features (like MFCCs) to create a codebook. Each utterance is then represented by a histogram of the nearest codebook entries, effectively creating a fixed-length representation. The authors explored various codebook sizes.

For classification, they employed Support Vector Machines (SVMs) with a linear kernel. Late fusion was used to combine predictions from different feature representations, with weights determined on development sets. To enhance robustness, they also explored building ensembles of classifiers by averaging predictions from models trained with different random seeds.

Datasets

The experiments were conducted on datasets from the INTERSPEECH 2019 Computational Paralinguistics Challenge (ComParE). The challenge comprised four sub-tasks, each with its own dataset:

  • Styrian Dialects Sub-Challenge: Identifying one of three south-eastern Austrian dialects.
  • Continuous Sleepiness Sub-Challenge: Estimating speaker sleepiness on a scale of 1 to 9 (regression task).
  • Baby Sounds Sub-Challenge: Categorizing vocalizations from infants aged 2 to 33 months.
  • Orca Activity Sub-Challenge: Detecting the presence of orca sounds in underwater recordings.

The paper mentions using development sets for parameter tuning and late fusion weight optimization, and final predictions were made on test sets.

Key Insights and Ideas

  • Fisher Vectors Show Promise: FV emerged as a strong contender, outperforming the baseline ComParE functionals in three out of the four sub-challenges on the development set. This suggests FV is a powerful and generalizable representation for paralinguistic tasks.
  • FV's Robustness: The authors highlight that FV appears to be a more robust approach compared to BoAW, which they suspect suffers from stochasticity due to its clustering step.
  • Late Fusion Benefits: Combining different feature representations through late fusion generally led to slight improvements, indicating that complementary information is captured by each method.
  • Task-Specific Performance: While FV generally performed well, the standard ComParE functionals were found to be more efficient for the Baby Sounds Sub-Challenge, suggesting that no single representation is universally optimal.
  • Ensembling for Robustness: Building ensembles of classifiers by averaging predictions from multiple runs (with different random seeds) further improved performance, particularly for the Sleepiness and Orca tasks, reinforcing the idea of enhancing robustness.

Key Results

  • Styrian Dialects: FV outperformed ComParE functionals on the development set. However, the combined model's test set performance (UAR of 29%) was below the baseline and even random guessing.
  • Continuous Sleepiness: FV consistently outperformed ComParE functionals. Late fusion and the inclusion of BoAW brought further slight improvements, with the final model achieving a Spearman's correlation coefficient of 0.387 on the test set.
  • Baby Sounds: ComParE functionals were more efficient on the development set. The best combined model achieved a test set UAR of 59.9%, a slight improvement over the baseline.
  • Orca Activity: FV showed comparable AUC scores to ComParE functionals. The hybrid model improved upon the baseline, and incorporating BoAW led to the highest AUC values (0.843 on dev, 0.879 on test).

Limitations and Future Work

  • Styrian Dialects Performance: The poor performance on the Styrian Dialects Sub-Challenge test set is a notable limitation, especially when compared to the baseline. The authors attribute the over-confidence of FV classifiers to potential overfitting.
  • BoAW Stochasticity: The paper points out the inherent stochastic nature of BoAW due to its clustering step, which can lead to performance variations between development and test sets.
  • Ensembling for Styrian Dialects: Due to time constraints, the authors were unable to perform ensemble experiments for the Styrian Dialects Sub-Challenge, which might have improved its performance.
  • Future Work: The authors suggest that further investigation into ensemble methods and exploring other general-purpose feature representations could be beneficial for paralinguistic tasks. They also emphasize the need for more robust methods to mitigate the stochasticity observed in BoAW.

Summary updated

BibTeX
@inproceedings{gosztolya2019using,
  author = {Gábor Gosztolya},
  title = {{Using Fisher Vector and Bag-of-Audio-Words Representations to Identify Styrian Dialects, Sleepiness, Baby \& Orca Sounds}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure