The BLZ Submission to the NIST 2011 LRE: Data Collection, System Development and Performance

Luis Javier Rodríguez-Fuentes, Mikel Peñagarikano, Amparo Varona, Mireia Díez, Germán Bordel, Alberto Abad, David Martínez González, Jesús Antonio Villalba López, Alfonso Ortega, Eduardo Lleida

Abstract

The NIST 2011 Language Recognition Evaluation (LRE) introduced significant challenges, including 24 target languages and a performance metric focused on the most difficult language pairs. The BLZ team—a collaboration between researchers in Bilbao, Lisboa, and Zaragoza—tackled these hurdles by fusing 13 distinct acoustic and phonotactic subsystems. This paper describes the ensemble architecture, the aggressive data collection strategy for low-resource languages, and the greedy search algorithm used for system fusion. Our results demonstrate that a diverse ensemble, combined with robust data augmentation, achieves state-of-the-art performance even on the most challenging language pairs.

Introduction

Language Recognition (LRE) is the automated process of identifying the language spoken in an audio segment. The NIST 2011 LRE significantly increased the task's complexity by including 24 target languages, nine of which were new and lacked extensive training data. Furthermore, NIST introduced a new primary metric: the average cost ($C_{avg}$) calculated over the 24 worst-performing language pairs. This shift required systems to be robust against "worst-case" scenarios, such as distinguishing between highly similar or data-scarce languages like Lao and Punjabi.

Methodology and System Fusion

The BLZ submission utilized a massive ensemble of 13 subsystems developed across three sites. These systems were categorized into two main types:

  1. Acoustic Systems: Focused on physical sound characteristics using iVectors, Joint Factor Analysis (JFA), and GMM-SVM approaches.
  2. Phonotactic Systems: Analyzed phoneme sequences (n-grams) derived from audio lattices produced by phone decoders.

To combine these diverse models, we employed Multiclass Discriminative Fusion. Given the $2^{13}-1$ possible combinations, an exhaustive search was impractical. Instead, a Greedy Search algorithm was used. Starting with the best individual system, we iteratively added the subsystem that provided the most significant performance gain.

Fusion analysis: $C_{avg}$ vs. number of fused subsystems.

As shown in the fusion analysis, the combination of multiple systems leads to a consistent decrease in both minimum and actual $C_{avg}$, demonstrating the high degree of complementarity between the acoustic and phonotactic models.

Data Collection Strategy

Data scarcity for the nine new target languages necessitated an extensive data-hunting mission. The BLZ team supplemented the limited NIST-provided data (approx. 25 minutes per new language) with:

  • Voice of America (VOA): Targeted fragments of new languages.
  • TV Archives: Audited broadcasts from sources like Arrabia TV (Maghrebi Arabic) and Telewizja Polska (Polish).
  • Existing Databases: Utilized COST-278 Broadcast News and previous NIST LRE datasets.
  • BLZ-train: A custom-built dataset specifically for the new target languages.

Results and Performance Analysis

The BLZ systems were among the top performers in the evaluation. The primary system (Pri) and contrastive systems (Con1-3) were evaluated across different audio durations (3s, 10s, and 30s).

Performance metrics for the 30s track across all pairs and the 24 worst pairs.

As detailed in the results table for the 30-second track, the best system (Con3) achieved a minimum $C_{avg}$ of 0.0071 for all pairs and 0.0509 for the 24 worst pairs. Performance significantly degraded for shorter segments, with $C_{avg}$ rising to ~0.024 for 10s and ~0.09 for 3s. Notably, the "24 worst pairs" metric was found to be highly correlated with the overall average cost, suggesting that systems that perform well generally also excel at difficult edge cases. Additionally, the team found that "zt-norm" normalization significantly improved results on 30-second clips but presented calibration challenges.

Conclusion

The BLZ submission highlights the power of ensemble methods and aggressive data collection in overcoming the challenges of low-resource language recognition. While the fusion strategy proved highly effective, we observed a mismatch between development and evaluation data, leading to calibration issues. Future work will focus on domain adaptation to bridge the gap between broadcast/web data and evaluation sets, as well as improving performance on very short (3-second) audio clips.

Summary updated

Abstract

This paper describes the most relevant features of a collaborative multi-site submission to the NIST 2011 Language Recognition Evaluation (LRE), consisting of one primary and three contrastive systems, each fusing different combinations of 13 state-of-the-art (acoustic and phonotactic) language recognition subsystems. The collaboration focused on collecting and sharing training data for those target languages for which few development data were provided by NIST, and on defining a common development dataset to train backend and fusion parameters and select the best fusions. Official and post-key results are presented and compared, revealing that the greedy approach applied to select the best fusions provided suboptimal but very competitive performance. Several factors contributed to the high performance attained by BLZ systems, including the availability of training data for low resource target languages, the reliability of the development dataset (consisting only of data audited by NIST), the diversity of modeling approaches, features and datasets in the systems considered for fusion, and the effectiveness of the search for optimal fusions.
BibTeX
@inproceedings{rodriguezfuentes2012blz,
  author = {Luis Javier Rodríguez-Fuentes and Mikel Peñagarikano and Amparo Varona and Mireia Díez and Germán Bordel and Alberto Abad and David Martínez González and Jesús Antonio Villalba López and Alfonso Ortega and Eduardo Lleida},
  title = {{The BLZ Submission to the NIST 2011 LRE: Data Collection, System Development and Performance}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-20},
}
Figure