The 2011 NIST language recognition evaluation

Craig S. Greenberg, Alvin F. Martin, Mark A. PrzybockiInterspeech 2012

TL;DR

The 2011 NIST Language Recognition Evaluation (LRE11) introduced a new evaluation metric focused on distinguishing difficult language pairs, reflecting the growing need for robust language identification in increasingly complex real-world scenarios. This evaluation highlighted that linguistically similar languages remain the hardest to differentiate, while also showing progress in distinguishing closely related dialects.

What Problem Does This Paper Address?

Automatic language recognition (ALR) systems are crucial for tasks like content filtering, routing calls, and information retrieval. However, as ALR technology advances, the challenge shifts from distinguishing vastly different languages to differentiating between closely related ones (e.g., Hindi/Urdu) or subtle dialectal variations (e.g., American English/Indian English). The NIST LRE series aims to track progress in this field. LRE11 specifically addressed the need for an evaluation metric that better reflects these real-world challenges by emphasizing the performance on difficult-to-distinguish language pairs, pushing research towards solving these finer-grained recognition problems.

Methods

LRE11 employed a novel evaluation metric designed to focus on challenging language pairs. Instead of a simple overall accuracy, the metric combined miss probabilities for each language within a pair using a cost model. This cost model was set to give equal weight to misses in either language of a pair, effectively penalizing systems that struggled with specific confusable pairs. Participants submitted their system outputs, which NIST then scored uniformly. The evaluation also included three different test segment durations (3, 10, and 30 seconds) to assess performance across varying amounts of speech data.

Datasets

LRE11 utilized a dataset comprising 24 languages, with nine being new to the LRE series. The data was sourced from two types: "found" narrowband conversational broadcast data and collected conversational telephone speech. For most languages, both data types were used. On average, each language had approximately 630 conversational telephone segments and 775 broadcast recordings from about 14 different sources.

Key Insights and Ideas

  • Linguistic Similarity is Key: The most challenging language pairs to distinguish were consistently those that are linguistically similar, such as Hindi/Urdu and Lao/Thai. This underscores that the frontier of ALR lies in fine-grained distinctions.
  • Pashto and Bengali are Broadly Confusable: These languages were found to be confusable with a wide range of other languages, indicating they present significant challenges across the board.
  • Progress in Dialect Distinction: The evaluation observed progress in distinguishing American English from Indian English, suggesting that ALR systems are improving at recognizing subtle dialectal variations.
  • Czech and Lao are Relatively Easy to Distinguish: Conversely, Czech and Lao were found to be easy to distinguish from most other languages in the evaluation, despite being part of confusable pairs with specific languages.
  • New Metric Drives Focus: The introduction of a metric that specifically targets difficult language pairs successfully shifted research focus towards these challenging distinctions.

Key Results

The paper presents overall performance measures for participating systems, visualized in bar charts. While specific quantitative results for individual systems are not detailed in a summary table, the figures illustrate the performance trends across different durations and language pairs. Notably, the Hindi/Urdu pair remained a persistent challenge, with only marginal improvement observed in the 3-second condition between LRE07 and LRE11. In contrast, the American-English/Indian-English pair showed consistent progress across all durations over successive evaluations.

Limitations and Future Work

The authors acknowledge that finding large, existing multilingual corpora for future evaluations is becoming increasingly difficult. Reverting to older data collection methods (e.g., single telephone calls per subject) could be prohibitively expensive. While the language pair focus is likely to continue, the exact timing and format of future LRE evaluations are yet to be determined. The paper also notes that the results are not to be construed as endorsements of any particular system or product.

Summary updated

BibTeX
@inproceedings{greenberg2012nist,
  author = {Craig S. Greenberg and Alvin F. Martin and Mark A. Przybocki},
  title = {{The 2011 NIST language recognition evaluation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure