Patrol Team Language Identification System for DARPA RATS P1 Evaluation

Pavel Matejka, Oldrich Plchot, Mehdi Soufifar, Ondrej Glembek, Luis Fernando D'Haro, Karel Veselý, Frantisek Grézl, Jeff Z. Ma, Spyros Matsoukas, Najim DehakInterspeech 2012 · OpenReview

Abstract

This paper details the "Patrol Team" submission for the first phase of the DARPA Robust Automatic Transcription of Speech (RATS) program. The team, a collaboration between Brno University of Technology, Raytheon BBN, and MIT CSAIL, developed a Language Identification (LID) system designed for extremely noisy and degraded radio communication environments. By fusing acoustic and phonotactic i-vector systems and employing neural network classifiers, the team achieved significant performance gains across various audio durations, particularly in the challenging 3-second condition.

Introduction

Traditional Language Identification (LID) research has primarily focused on relatively clean telephone or broadcast data. However, real-world signals in defense and emergency contexts are often transmitted over low-quality radio channels, characterized by non-additive noise and "push-to-talk" (PTT) clipping. The DARPA RATS program was established to address these challenges, requiring the identification of five target languages—Dari, Arabic Levantine, Urdu, Pashto, and Farsi—from audio segments as short as 3 seconds recorded over eight highly degraded communication channels.

System Architecture

The Patrol Team utilized a fusion-based approach, integrating four distinct subsystems to maximize robustness:

  1. Acoustic JFA: A Joint Factor Analysis model using shifted-delta-cepstra (SDC) features, pre-processed with Wiener filtering to suppress channel noise.
  2. i-vector NN (BUT): An acoustic i-vector extractor where classification is performed by a three-layer Neural Network (NN), which proved more effective than standard logistic regression for noisy data.
  3. i-vector NN (BBN): This system utilized Language-Dependent Background Models (LDBM) and was trained on 20-second audio "chunks" to better simulate the short-duration segments encountered during evaluation.
  4. Phonotactic (PHN-CZ): A system that converts speech into phoneme "soft-counts" using a Czech phone recognizer, which are then modeled using i-vectors.

The final system output is a weighted fusion of these four subsystems, calibrated to produce log-likelihood ratios.

Key Technical Insights

Several strategies were critical to the system's success in the RATS environment. Applying a Wiener filter as a pre-processor provided a consistent boost in performance, particularly for acoustic models. Furthermore, the use of Neural Networks for classification and the adoption of Language-Dependent Background Models (LDBM) were pivotal.

Comparison of UBM and LDBM performance across different chunk sizes and test durations.

As shown in the table above, the transition from a standard Universal Background Model (UBM) to an LDBM, combined with appropriate chunking (20s), led to substantial reductions in $C_{avg}$, particularly for shorter audio segments.

Evaluation and Results

The system was evaluated using the average cost of detection ($C_{avg}$) across four test durations: 120s, 30s, 10s, and 3s.

Extracted table from page 4 of Patrol Team Language Identification System for DARPA RATS P1 Evaluation

The results demonstrate the power of system fusion. On the 120s development set, the fusion system achieved a $C_{avg}$ of 0.83%, nearly halving the error of the best individual system (i-vector BUT at 1.60%). While performance naturally degrades as audio duration decreases, the fusion system maintained a competitive $C_{avg}$ of 17.45% even in the extreme 3-second evaluation condition.

Conclusion

The Patrol Team's work confirms that state-of-the-art LID techniques like i-vectors remain highly effective in extreme noise conditions when properly adapted. The combination of Wiener filtering, neural network classification, and the fusion of acoustic and phonotactic models provides a robust framework for identifying languages in degraded communication channels. Future work will focus on further improving performance for ultra-short segments and better handling of non-speech regions in radio transmissions.

Summary updated

BibTeX
@inproceedings{matejka2012patrol,
  author = {Pavel Matejka and Oldrich Plchot and Mehdi Soufifar and Ondrej Glembek and Luis Fernando D'Haro and Karel Veselý and Frantisek Grézl and Jeff Z. Ma and Spyros Matsoukas and Najim Dehak},
  title = {{Patrol Team Language Identification System for DARPA RATS P1 Evaluation}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=gGceXJIzN6f},
}
Figure