Study of different backends in a state-of-the-art language recognition system

Mikel Penagarikano, Amparo Varona, Mireia Diez, Luis Javier Rodriguez-Fuentes, German BordelInterspeech 2012

TL;DR

This paper investigates the impact of various "backend" models on the performance of state-of-the-art spoken language recognition (SLR) systems. They found that a 2-component Gaussian Mixture Backend (GMB) and a Discriminative Gaussian Backend (DGB) offered the best performance, with GMB being a more robust choice. Additionally, they demonstrated that simple techniques like score normalization (T-norm/ZT-norm) and filtering out high-scoring development signals can significantly improve accuracy.

What Problem Does This Paper Address?

Spoken Language Recognition (SLR) systems aim to identify the language being spoken in an audio recording. A typical SLR system involves several stages, including feature extraction, scoring by individual language models, and then a "backend" stage. This backend plays a crucial role: it can map scores from languages the system was trained on to the specific target languages needed for a task, and it also calibrates and adapts the raw scores to make them more reliable. The challenge is that there are many different backend approaches, and their effectiveness can vary significantly depending on the overall system architecture and the data used. This paper seeks to systematically compare several well-established and newer backend techniques to understand their relative strengths and weaknesses in a state-of-the-art SLR system.

Methods

The authors compare several backend approaches:

  • Generative Gaussian Backend (GB): A standard approach that models language scores using multivariate normal distributions.
  • Discriminative Gaussian Backend (DGB): An extension of GB that further optimizes parameters to maximize the Maximum Mutual Information (MMI) criterion.
  • Logistic Regression Backend (LR): Uses regularized multiclass logistic regression to learn an affine transform for score calibration.
  • Fully-Bayesian Gaussian Backend (FBGB): Integrates over model parameters rather than using point estimates, aiming for better robustness.
  • Gaussian Mixture Backend (GMB): Augments the Gaussian model with a mixture of Gaussians, allowing for more flexible modeling of score distributions.

They also investigate the impact of:

  • Score Normalization: Specifically, Z-norm and T-norm techniques, which aim to reduce variability in scores caused by environmental factors.
  • Development Signal Filtering: Removing the highest-scoring development signals, hypothesizing that these might correspond to repeated speakers and could lead to overfitting.

The experiments were conducted using a sophisticated SLR system that fuses the outputs of five subsystems: a Linearized Eigenchannel GMM (LE-GMM), an iVector subsystem, and three phone-lattice SVM subsystems.

Datasets

The experiments were carried out on the NIST 2009 LRE (Language Recognition Evaluation) database.

  • Training and Development Data: Included conversational telephone speech from previous LREs (Call-Friend Corpus, OHSU Corpus, NIST 2007 LRE development corpus) and narrow-band speech segments from Voice Of America (VOA) broadcast news recordings.
  • Evaluation Data: The NIST 2009 LRE evaluation corpus, specifically the 30-second, closed-set condition.

A total of 66 languages/dialects were mapped to target languages or Out-Of-Set (OOS). The training dataset comprised 43,278 segments totaling 2,286 hours, while the development dataset had 13,269 segments totaling approximately 380 hours.

Key Insights and Ideas

  • GMB and DGB Shine: The Gaussian Mixture Backend (GMB) and Discriminative Gaussian Backend (DGB) consistently outperformed other backends across various subsystems. GMB is highlighted as a more conservative but effective choice.
  • GMB's Component Count Matters: While GMB generally improved performance, the number of mixture components was important. A 2-component GMB provided the best balance, with performance degrading as the number of components increased beyond that.
  • Filtering is Surprisingly Effective: Removing the top-scoring development signals (around 1200 out of 13269) led to significant performance improvements, particularly for the iVector subsystem. This suggests that uncontrolled development sets can indeed contain problematic repeated speakers.
  • Score Normalization's Nuances: T-norm and ZT-norm showed varying effects depending on the subsystem. While they could improve performance, their impact was not uniform, and sometimes they even degraded results for certain acoustic subsystems.
  • FBGB vs. GB: The Fully-Bayesian Gaussian Backend (FBGB) offered only a marginal improvement over the standard Generative Gaussian Backend (GB). This might indicate that the development set was sufficiently large for GB to be well-estimated, mitigating the need for Bayesian regularization.

Key Results

  • The best overall performance was achieved by the Gaussian Mixture Backend (GMB), resulting in an Equal Error Rate (EER) of 1.25%.
  • This represents a 23% relative improvement compared to the baseline system using the Generative Gaussian Backend (GB) without any normalization or filtering, which achieved an EER of 1.62%.
  • Filtering out the 1200 highest-scoring development signals improved the iVector subsystem's performance.
  • The Russian phonotactic subsystem (Phon-RU) was the best-performing individual subsystem, followed by the Dot-Scoring acoustic subsystem.

Limitations and Future Work

  • Logistic Regression Performance: The multiclass Logistic Regression backend did not achieve the expected results, possibly due to issues with its estimation algorithm.
  • MMI Tuning for DGB: The Discriminative Gaussian Backend (DGB) achieved good performance but required careful tuning of the MMI factor and the number of training iterations, with a risk of overfitting.
  • Future Work: The paper implicitly suggests further exploration into optimizing score normalization techniques and investigating more advanced filtering strategies for development data. The authors also note that the GMB is a more conservative alternative to DGB, implying that further research could explore ways to achieve DGB's performance with GMB's robustness.

Summary updated

BibTeX
@inproceedings{penagarikano2012study,
  author = {Mikel Penagarikano and Amparo Varona and Mireia Diez and Luis Javier Rodriguez-Fuentes and German Bordel},
  title = {{Study of different backends in a state-of-the-art language recognition system}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure