Styrian Dialect Classification: Comparing and Fusing Classifiers Based on a Feature Selection Using a Genetic Algorithm

Thomas Kisler, Raphael Winkelmann, Florian SchielInterspeech 2019

TL;DR

This paper tackles the challenge of classifying Styrian dialects using acoustic features, which often suffer from high dimensionality. By employing a Genetic Algorithm (GA) for feature selection, they significantly reduce the feature space and improve classification performance across various models, with a fusion of the top classifiers achieving the best results.

What Problem Does This Paper Address?

Automatic dialect classification aims to identify regional variations in speech. While many acoustic features can be extracted from speech signals, the sheer volume of these features (over 10,000 in this case) can overwhelm classifiers, leading to decreased performance due to noise and irrelevant information. This is particularly challenging for phonetically similar dialects, like the East Bavarian dialects studied here. The paper addresses this "curse of dimensionality" by proposing an effective feature selection method to identify the most informative features for dialect classification.

Methods

The core of this paper is the application of a Genetic Algorithm (GA) for feature selection. The GA works by evolving a population of candidate feature subsets. Each subset is represented as a "chromosome" (a binary string where '1' indicates a feature is included and '0' excludes it). The fitness of each chromosome is evaluated using a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel on a development dataset. This process iteratively combines and mutates the best-performing feature subsets to discover an optimal, reduced set.

Beyond the standard ComParE features, the authors also introduce an additional per-phoneme feature set. This set is derived by segmenting speech into phonemes and extracting features for each, aiming to capture finer-grained pronunciation differences. They also create a combined feature set by merging the selected features from both the challenge and additional feature sets.

Finally, they compare the performance of five different classifiers (SVM, DNN, Lasso, GBM, RF) on these reduced feature sets. They also explore a fusion strategy, specifically a 3-best majority vote, where the predictions of the three top-performing classifiers are combined to make the final decision.

Datasets

The study utilizes data from the STYRIALECTS corpus, specifically focusing on three phonetically close East Bavarian dialects from Styria, Austria. The challenge provided three official feature sets:

  • ComParE Features: A large set (6373 features) of long-term functionals based on low-level descriptors (LLDs).
  • BoAW Features: Bag-of-Audio-Words features, an unsupervised representation.
  • AUDEEP Features: Features extracted using recurrent sequence-to-sequence autoencoders.

These were combined into a Challenge Feature Set of 12,470 features.

The authors also created:

  • Additional Feature Set: A per-phoneme feature set (742 features) derived using phonetic segmentation and labeling.
  • Combined Feature Set: A fusion of the GA-selected features from both the Challenge and Additional Feature Sets (429 features).

All feature sets were standardized based on the mean and standard deviation of the training set.

Key Insights and Ideas

  • Feature Selection is Crucial: The paper strongly demonstrates that reducing a high-dimensional feature space is essential for improving classification performance, especially for phonetically similar dialects.
  • GA as an Effective Feature Selector: The Genetic Algorithm proved to be a powerful tool for identifying a subset of features that significantly boosted classifier performance, outperforming the baseline.
  • Per-Phoneme Features Offer Value: The additional per-phoneme feature set, while smaller, contributed valuable information, especially when combined with global features.
  • Fusion Outperforms Individual Classifiers: Combining the predictions of the top three classifiers (a 3-best majority vote) consistently yielded the best results, highlighting the benefit of ensemble methods.
  • Generalization Gap: A significant drop in performance was observed between the development and test sets, suggesting that the GA-selected features might have overfit to the specific validation set and classifier used during the selection process.

Key Results

  • On the Challenge Feature Set, the GA selected 323 features, leading to a maximum Unweighted Average Recall (UAR) of 0.6523 on the development set. The SVM, which was also used for fitness evaluation, achieved the best individual performance.
  • For the Additional Feature Set, the GA identified 149 features, resulting in a best UAR of 0.5609. Here, a fusion of the top three classifiers (Lasso, SVM, DNN) achieved the highest UAR.
  • The Combined Feature Set, after GA selection, resulted in 429 features. While individual classifiers performed well, the 3-best fusion achieved a UAR of 0.7055, which was the best performance across all tested subsets and methods.
  • On the test set, performance dropped significantly across all configurations, with the best fusion achieving a UAR of 0.4129. This was still comparable to or better than the baseline reported in previous work, except for one specific result.

Limitations and Future Work

The authors acknowledge a significant limitation: the generalization gap between the development and test sets. They attribute this to the GA's feature selection process being highly adapted to the specific validation set and the fitness classifier (SVM). This implies that the selected features might have overfit to the development data.

Future work could address this by:

  • Employing n-fold Cross-Validation (CV) during the GA's fitness evaluation to ensure better generalization and account for speaker variability.
  • Investigating the combination of per-phoneme features with global features more thoroughly, potentially incorporating methods from related work on speaker localization.
  • Further exploring ensemble methods and the impact of different fusion strategies.
  • Investigating the performance of tree-based classifiers more deeply, as they generally lagged behind other methods in this study.

Summary updated

BibTeX
@inproceedings{kisler2019styrian,
  author = {Thomas Kisler and Raphael Winkelmann and Florian Schiel},
  title = {{Styrian Dialect Classification: Comparing and Fusing Classifiers Based on a Feature Selection Using a Genetic Algorithm}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure