English spoken term detection in multilingual recordings

Petr Motlícek, Fabio Valente, Philip N. GarnerInterspeech 2010 · OpenReview

Abstract

Spoken Term Detection (STD) systems typically assume a monolingual input, leading to performance degradation in multilingual environments where "out-of-language" (OOL) speech is incorrectly mapped to target language terms. This paper proposes a robust "filter-then-search" framework that identifies OOL segments using confidence measures derived from speech recognizer word lattices. By filtering non-English speech and incorporating text from presentation slides as a prior, the system achieves a relative error reduction of over 50%.

Introduction

Searching for specific keywords in spoken audio is a core task in information retrieval. However, real-world recordings, such as academic lectures in multilingual regions, often switch between languages. When an English-only STD system encounters non-English speech (e.g., French or Italian), it attempts to force-fit these sounds into English phonetic models, resulting in a high rate of False Alarms (FAs). This research addresses the challenge of distinguishing target language speech from OOL speech without requiring complex recognizers for every potential language.

Methodology

The researchers implemented a two-stage pipeline to enhance STD performance in multilingual settings.

Out-Of-Language (OOL) Detection

The primary filter relies on the observation that a speech recognizer becomes "confused" when processing a language it wasn't trained for. This confusion is quantified using several Confidence Measures (CMs) extracted from word lattices. Among these, Word Lattice Width ($W_{lat}$)—the number of active arcs in the lattice at a given time—proved most effective. To reduce noise, a temporal filter (typically a 3-second window) is applied to smooth the OOL scores.

Multimodal Boosting

For domain-specific content like lectures, external metadata such as PowerPoint slides often contain the very terms users are searching for. The system incorporates this by using slide text as a prior. If a search term appears on the slides, its detection score is boosted by a multiplicative constant ($c$), effectively lowering the threshold for high-confidence matches.

Key Results

The effectiveness of the OOL detection module is a critical component of the system's success. As shown in the table below, increasing the temporal window significantly improves the Equal Error Rate (EER) of OOL detection.

OOL detection performance across different measures and window lengths.

The integration of the OOL filter into the STD pipeline led to dramatic improvements. On the Klewel lecture dataset, which contains a mix of English, French, and Italian, the baseline STD system had an EER of 8.1%. Applying the automated OOL filter reduced this to 4.0%, nearly matching the 3.5% EER achieved with manual language segmentation.

DET curves showing STD performance improvements with OOL filtering.

Furthermore, the addition of slide-based metadata provided an additional 15% relative improvement in performance, demonstrating the value of multimodal information in specialized search tasks.

Conclusion

This work demonstrates that OOL detection based on recognizer uncertainty is a powerful tool for improving Spoken Term Detection in multilingual recordings. By filtering out non-target speech before searching, systems can significantly reduce false alarms. Future work will explore "soft" integration of OOL scores and finer-grained temporal alignment of external metadata like presentation slides.

Summary updated

BibTeX
@inproceedings{motlicek2010english,
  author = {Petr Motlícek and Fabio Valente and Philip N. Garner},
  title = {{English spoken term detection in multilingual recordings}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=-PAx_tKWgS},
}
Figure