Rescoring Keyword Search Confidence Estimates with Graph-Based Re-Ranking Using Acoustic Word Embeddings

Anna Piunova, Eugen Beck, Ralf Schlüter, Hermann NeyInterspeech 2019

TL;DR

This paper introduces a novel approach to improve keyword search (KWS) in audio by replacing traditional acoustic similarity measures with Acoustic Word Embeddings (AWEs) within a graph-based re-ranking framework. This method significantly boosts KWS performance, especially for challenging out-of-vocabulary (OOV) queries, by better capturing semantic and acoustic similarities between potential matches.

What Problem Does This Paper Address?

Searching for specific spoken terms within large audio archives (like lecture recordings or broadcast news) is a crucial but challenging task. Existing Keyword Search (KWS) systems often rely on Automatic Speech Recognition (ASR) to generate lattices of possible word sequences, from which potential "hits" are extracted and ranked by confidence scores. However, these scores can be unreliable, especially in low-resource languages or for out-of-vocabulary (OOV) words, leading to missed detections or false alarms. To address this, researchers use post-processing techniques like graph-based re-ranking, which builds a graph of potential hits and rescores them based on their similarity to other high-confidence hits. The standard method for measuring this acoustic similarity between hits is Dynamic Time Warping (DTW). This paper tackles the limitation of DTW, which can be sensitive to acoustic variations, by proposing the use of Acoustic Word Embeddings (AWEs) as a more robust similarity measure.

Methods

The core of this work is the integration of Acoustic Word Embeddings (AWEs) into a graph-based re-ranking framework. The authors adapt a previously proposed graph-based re-ranking algorithm, which builds a similarity graph where nodes are potential KWS hits and edges represent acoustic similarity. Instead of using Dynamic Time Warping (DTW) to compute the distance between acoustic features of hits, they now use the cosine distance between their corresponding AWEs.

They explore three types of AWE models:

  • Classifier Model: A supervised approach using bidirectional LSTMs (biLSTMs) trained to classify word identities from acoustic sequences. This model requires significant labeled data.
  • Siamese Network (Weakly-supervised): This model learns embeddings by being trained to maximize the distance between embeddings of different words while minimizing the distance between embeddings of the same word. It uses word-pair information rather than exact word identities.
  • Sequence-to-Sequence Autoencoder (Unsupervised): This model learns representations by encoding an acoustic sequence and then decoding it to reconstruct the original sequence, without requiring any explicit word labels.

Additionally, they enhance the detection lists not only with positive query exemplars (ground truth detections) but also with negative query exemplars (acoustically similar words that are likely false alarms). This aims to further refine the re-ranking process by helping to distinguish true positives from false positives.

Datasets

The experiments were conducted on audio corpora from the IARPA Babel Program for two low-resource languages: Pashto and Mongolian.

  • Training Data: Approximately 50-60 hours of transcribed telephone conversational speech for each language was used to train the AWE models. The training vocabularies were substantial (14,436 for Pashto, 24,054 for Mongolian).
  • Evaluation Data: A 10-hour development set for each language was used for evaluation, containing 2,065 search queries for Pashto and 2,404 for Mongolian.

The paper also mentions the use of different KWS systems and underlying ASR systems, including those using a "transcription lexicon" (built from provided training data) and a "web lexicon" (extracted via web crawling), with some systems further rescoring lattices with an LSTM language model.

Key Insights and Ideas

  • AWEs as a Superior Similarity Measure: The central idea is that AWEs, particularly those from classifier models, are more robust to acoustic variations than DTW, leading to better discrimination between correct detections and false alarms. This is because AWEs are trained to capture the underlying semantic or acoustic identity of words.
  • Classifier Embeddings Shine for OOV: Classifier-based AWEs showed the most significant improvements, especially for out-of-vocabulary (OOV) queries, suggesting that explicit word identity training helps generalize to unseen words.
  • Negative Exemplars Improve Robustness: Augmenting detection lists with negative query exemplars (acoustically similar but incorrect words) further boosted performance, indicating that explicitly teaching the system what not to match is as important as teaching it what to match.
  • Language and Lexicon Dependence: The effectiveness of different AWE models varied across languages and KWS system setups (e.g., transcription vs. web lexicon), highlighting the need for careful model selection and tuning.
  • Unsupervised Models Lag for Core Task: While unsupervised AWEs (like sequence-to-sequence autoencoders) performed well on OOV queries in certain setups, they generally underperformed compared to supervised or weakly-supervised methods for the main KWS task.

Key Results

The proposed graph-based re-ranking using AWEs, particularly the Classifier model, significantly improved the Maximum Term Weighted Value (MTWV) metric.

  • Pashto: Relative improvements of 3.0% to 7.5% over the baseline KWS result were observed. The Classifier AWEs improved over DTW-based rescoring by 0.5% on in-vocabulary (IV) queries and 2.8% on OOV queries.
  • Mongolian: A 3.0% relative MTWV increase was achieved over the baseline KWS result. Improvements over DTW were 0.3% on IV and 0.8% on OOV queries.
  • Negative Exemplars: Adding negative exemplars consistently improved MTWV across most setups, with the exception of the sequence-to-sequence autoencoder on the transcription lexicon.

Limitations and Future Work

  • Model Selection: The paper notes that the optimal AWE model and training strategy can depend on the specific language, KWS system, and query type (IV vs. OOV).
  • Combination Strategies: The authors suggest that combining different models for IV and OOV queries could be a promising avenue for further improvement.
  • Unsupervised Model Performance: While unsupervised models showed some promise for OOV, their overall performance lagged behind supervised methods, suggesting room for improvement in unsupervised AWE learning for KWS.
  • Negative Exemplar Sampling: The optimal number of negative exemplars to sample was found to be variable and dependent on the setup, indicating that more sophisticated sampling strategies might be beneficial.

Summary updated

BibTeX
@inproceedings{piunova2019rescoring,
  author = {Anna Piunova and Eugen Beck and Ralf Schlüter and Hermann Ney},
  title = {{Rescoring Keyword Search Confidence Estimates with Graph-Based Re-Ranking Using Acoustic Word Embeddings}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure