Subword RNNLM Approximations for Out-Of-Vocabulary Keyword Search

Mittul Singh, Sami Virpioja, Peter Smit, Mikko KurimoInterspeech 2019

TL;DR

This paper tackles the challenge of searching for spoken keywords that weren't in the original speech recognition training data (OOVs). They propose a novel method to approximate powerful but slow Recurrent Neural Network Language Models (RNNLMs) into faster n-gram models, specifically tailored for subword units. This approach significantly improves the accuracy of Out-of-Vocabulary keyword search, especially when using character-level subwords.

What Problem Does This Paper Address?

Keyword Search (KWS) systems aim to find specific words or phrases within audio recordings. A major hurdle arises when the target keywords are "Out-of-Vocabulary" (OOV) – meaning they weren't part of the vocabulary used to train the underlying Automatic Speech Recognition (ASR) system. Traditional ASR systems simply fail to recognize these words. To address this, researchers have explored using subword units (like characters or morphemes) instead of whole words, allowing the system to potentially recognize unseen words by composing them from known subword units. While subword-based language models (LMs) help, standard n-gram LMs still struggle with data sparsity for these subword sequences. More powerful Recurrent Neural Network Language Models (RNNLMs) can alleviate sparsity but are too computationally expensive for the initial "first-pass" decoding stage of KWS. This paper aims to bridge this gap by creating efficient approximations of RNNLMs that can be used in the first-pass decoding for improved OOV keyword search.

Methods

The core of this paper lies in approximating RNNLMs to n-gram LMs for subword units. They build upon the "probability-conversion" (PC) method but introduce a novel "n-gram-growing" algorithm.

  • Baseline RNNLM Approximation (PC Method): The standard PC method approximates RNNLMs by marginalizing over observed sentence histories. However, it can be inefficient for higher-order n-grams and may miss rare but important subword contexts.
  • Proposed RNNLM Approximation (Ours): The authors propose a modified approach that focuses on storing only relevant probabilities from the RNN output. Specifically, they consider probabilities for contexts where the subword sequence is either observed in the corpus or is among the top-K most probable next subwords. This significantly reduces computational and storage requirements.
  • N-gram Growing Algorithm: To further enhance the approximation and capture longer contexts (crucial for morphologically rich languages and longer OOV words), they integrate their approximation method into an n-gram-growing algorithm. This algorithm iteratively expands n-grams, selecting important contexts based on a minimum description length criterion. This allows for building longer-span n-gram LMs from the approximated RNNs.
  • Interpolation: They also explore interpolating their approximated RNNLMs (both PC-based and their own) with traditional Kneser-Ney smoothed n-gram models (KNV) to combine the strengths of both approaches.

Datasets

The experiments were conducted on Arabic and Finnish datasets, chosen for their morphological richness, which leads to a higher OOV rate (around 2%).

  • Arabic:
    • ASR Training: 1,200 hours of Aljazeera television programs (MGB-2 challenge).
  • ASR Testing: MGB-2 development set (8 hours).
  • LM Training: 130 million tokens from Aljazeera's website.
  • Finnish:
    • ASR Training: 1,500 hours from Speecon, Speechdat, and the parliament corpus.
  • ASR Testing: 5 hours of broadcast news from Yle.
  • LM Training: Not explicitly detailed, but implied to be from the Finnish Text Collection (143M tokens).

The paper also highlights the creation of specific OOV keyword lists for evaluation, filtering out single-character OOVs or those with unseen characters.

Key Insights and Ideas

  • Subword Units are Crucial for OOVs: The paper reinforces that using subword units (especially characters) is essential for handling OOV keywords, as they can be constructed from known parts.
  • RNNLMs Offer Potential but Need Approximation: While RNNLMs are powerful for modeling language and handling sparsity, their computational cost makes them impractical for first-pass decoding. Approximation is key.
  • Character-Level Subwords Outperform Morphs: For OOV keyword search, character-level subwords generally yield better performance than morph-based subwords. This is likely because OOVs tend to be longer, and characters offer a finer granularity for representation.
  • Novel Approximation Method is Effective: The proposed RNNLM approximation method, which focuses on top-K probabilities and integrates with an n-gram-growing algorithm, shows significant improvements over baseline PC methods.
  • Longer Contexts Matter: The n-gram-growing aspect of their method, enabling longer contexts, is shown to be beneficial, especially for morphologically rich languages.
  • Interpolation Boosts Performance: Combining the approximated RNNLMs with traditional n-gram models through interpolation further enhances KWS performance.

Key Results

  • Improved OOV Detection: The proposed interpolated models (KNV+RNNV) consistently outperform the baseline KNV model for both Arabic and Finnish KWS tasks.
  • Character-Level Dominance: Single-character subword models, particularly with the proposed RNNV approximation and interpolation, achieve the best overall performance.
  • Significant Gains: Interpolated models show larger improvements on characters (up to 15.1% on Arabic) compared to morphs (up to 7.2% on Arabic).
  • RNNV vs. RNN5: The proposed RNNV approximation generally performs better than the PC-based RNN5 approximation, especially when using characters.
  • Exceeding Benchmarks: The achieved Maximum Term Weighted Values (MTWV) on Arabic and Finnish KWS tasks were higher than the IARPA BABEL Program's aim of 0.5 MTWV.

Limitations and Future Work

  • Approximation vs. Proper Distribution: The proposed approximation method (Equation 3) is noted as not being a proper probability distribution, which can lead to higher missing probability mass when creating back-off LMs, especially for small K values.
  • Parameter Sensitivity: The performance of the RNNV approximation is sensitive to the choice of parameters like K (top-K step) and the n-gram context size (n). Finding optimal values is crucial.
  • Future Work: The authors suggest investigating the effect of rescoring on this KWS setup and exploring performance on under-resourced scenarios, as prescribed by the IARPA BABEL program.

Summary updated

BibTeX
@inproceedings{singh2019subword,
  author = {Mittul Singh and Sami Virpioja and Peter Smit and Mikko Kurimo},
  title = {{Subword RNNLM Approximations for Out-Of-Vocabulary Keyword Search}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure