The EHU systems for the NIST 2011 language recognition evaluation

Mikel Penagarikano, Amparo Varona, Luis Javier Rodriguez-Fuentes, Mireia Diez, German BordelInterspeech 2012

TL;DR

This paper details the University of the Basque Country's (EHU) systems for the NIST 2011 Language Recognition Evaluation (LRE). They achieved strong results by fusing acoustic and phonotactic subsystems, demonstrating the power of combining different speech analysis techniques. A key takeaway is the challenge of robust language identification, especially with new languages and shorter speech segments, highlighting the importance of comprehensive training data.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately identifying spoken languages from audio. This is crucial for applications like automatic content analysis, multilingual call centers, and surveillance. The NIST 2011 LRE introduced specific difficulties: a larger set of target languages, including many previously unused ones, and a new performance metric that heavily penalized systems struggling with even a few language pairs. This meant that simply performing well on average wasn't enough; systems needed to be robust across all languages and durations, especially for short utterances (3-10 seconds), which are notoriously harder to classify.

Methods

The EHU team developed a multi-system approach, fusing five component subsystems:

  • Acoustic Subsystems:
    • Linearized Eigenchannel GMM (LE-GMM) / Dot-Scoring: This system uses Gaussian Mixture Models (GMMs) adapted with MAP estimation. It employs a linearized scoring procedure based on channel-compensated means and first-order Baum-Welch statistics.
  • iVector Subsystem: This system leverages i-vectors, a popular technique for speaker and language recognition, to represent utterances in a low-dimensional subspace capturing speaker and channel variability. They adapted a standard i-vector approach, though with a slight deviation in channel compensation timing due to hardware constraints.
  • Phonotactic Subsystems: These systems analyze sequences of phonetic units.
  • Phone-Lattice-SVM: Three such subsystems were built using publicly available phone decoders (from Brno University of Technology - BUT) for Czech, Hungarian, and Russian. These decoders generate phone lattices, from which phone n-gram counts are extracted. These counts are then used to train Support Vector Machines (SVMs) for classification.
  • Fusion: The core of their strategy was fusing these subsystems. They used a multiclass discriminative fusion approach, specifically linear logistic regression, to combine the scores from the acoustic and phonotactic systems. This fusion was performed separately for each nominal duration (3s, 10s, 30s). They also experimented with a Gaussian backend (with optional zt-norm) to process the subsystem scores before fusion.

Datasets

  • Training Data:
    • Conversational Telephone Speech (CTS) from previous LREs.
  • Narrow-band speech segments from Voice-of-America (VOA) broadcasts.
  • For newly added languages in NIST 2011 LRE:
  • A dedicated development dataset (lre11) provided by NIST (100 30-second segments per language, except Lao).
  • Additional data collected from various sources, including broadcast news, telephone speech databases (LDC), and TV websites.
  • For Punjabi, only a very limited amount of training data was available.
  • Development Data:
    • A combined development set was created to tune systems robustly. This included:
  • Evaluation sets from NIST 2007 and 2009 LREs (for languages also present in NIST 2011).
  • The lre11-dev subset (derived from the lre11 dataset).
  • This development set was split into three subsets based on nominal duration: dev30 (30s), dev10 (10s), and dev03 (3s).

Key Insights and Ideas

  • Complementarity is Key: The paper strongly reinforces the idea that combining acoustic and phonotactic approaches yields superior results. Acoustic systems capture spectral characteristics, while phonotactic systems analyze speech sound sequences, offering complementary information.
  • Fusion is Crucial for Robustness: The fusion of multiple subsystems, rather than relying on a single best system, was essential for achieving competitive performance, especially given the diverse and challenging nature of the NIST 2011 LRE.
  • Data Sparsity is a Major Hurdle: The limited availability of training and development data for newly introduced languages posed a significant challenge, increasing the risk of overtraining and impacting robustness, particularly for short utterances.
  • Short Utterances are Difficult: Classifying languages from very short segments (3-10 seconds) remains a substantial challenge, with performance significantly degrading compared to longer segments.
  • New Metric's Impact: The new performance metric, focusing on the 24 worst language pairs, meant that even a single poorly performing language could drastically affect the overall score, emphasizing the need for uniform performance across all languages.

Key Results

  • The EHU systems achieved results among the best submitted to the NIST 2011 LRE, particularly strong in the 30-second track.
  • No significant performance differences were observed among the four developed EHU systems (one primary, three contrastive), suggesting their fusion strategies were robust.
  • For 3-second segments, the third contrastive system outperformed the others, likely due to a more reliable development dataset used for its backend and fusion parameters.
  • Post-key experiments showed that the Russian phonotactic subsystem (Phone-RU) performed best among individual components (act-Cavg × 100 = 13.76).
  • Fusion of phonotactic subsystems alone achieved a good result (act-Cavg × 100 = 10.13), but the full fusion including acoustic subsystems provided a further 12% improvement.
  • The fusion of the Phone-HU and iVector subsystems showed the highest relative improvement (over 25%), highlighting the synergy between phonotactic and acoustic methods.

Limitations and Future Work

  • Poor Calibration: A significant limitation noted was the poor calibration achieved across all systems, indicated by large differences between minimum and actual Cavg. This suggests a potential mismatch between development and evaluation datasets or insufficient development data for some languages.
  • Data Limitations for New Languages: The paper acknowledges that the limited development data for newly added languages made them prone to overtraining and robustness issues.
  • ZT-Norm Impact: Applying zt-norm before the backend did not significantly affect performance, suggesting its utility might be limited in this specific setup or with the available data.
  • Future Work: The authors suggest further investigation into the optimal combination of subsystems, as demonstrated by their greedy selection strategy. The paper implies that continued efforts in data collection and augmentation for under-represented languages would be beneficial.

Summary updated

BibTeX
@inproceedings{penagarikano2012ehu,
  author = {Mikel Penagarikano and Amparo Varona and Luis Javier Rodriguez-Fuentes and Mireia Diez and German Bordel},
  title = {{The EHU systems for the NIST 2011 language recognition evaluation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure