Exploiting discriminative point process models for spoken term detection

Atta Norouzian, Aren Jansen, Richard C. Rose, Samuel ThomasInterspeech 2012

TL;DR

This paper introduces a novel way to improve spoken term detection (STD) by using discriminative point process models (DPPMs) to rescore candidate word occurrences found in ASR lattices. By leveraging a new distance metric for phonetic event patterns, they achieve performance comparable to state-of-the-art systems and demonstrate that their DPPM scores are complementary to existing methods, leading to further improvements when fused.

What Problem Does This Paper Address?

Spoken Term Detection (STD) aims to find specific words within large audio archives. The standard approach involves using a large vocabulary continuous speech recognition (LVCSR) system to generate "lattices" – essentially, multiple hypotheses of word sequences with associated probabilities. While these lattices help mitigate the high word error rates of single-best ASR outputs, they often struggle to perfectly capture the natural variations in speech pronunciation and timing. This paper addresses the challenge of improving the accuracy of STD by developing a more robust acoustic modeling approach that can better verify candidate word occurrences identified by these lattices. They propose moving beyond traditional Hidden Markov Model (HMM) and Gaussian Mixture Model (GMM) based acoustic models to a point process framework, which models speech as a sequence of discrete phonetic events.

Methods

The core innovation lies in the application of Discriminative Point Process Models (DPPMs) for spoken term detection. Here's a breakdown:

  • Point Process Modeling: Instead of traditional acoustic models, this approach represents speech as a series of discrete "events" (e.g., phonetic occurrences) over time. These events form "point patterns."
  • Whole Word Models: The system trains models for entire words, not just individual phonemes. These word models learn to recognize the temporal patterns of phonetic events that constitute a specific word.
  • Discriminative Training: DPPMs are trained discriminatively, meaning they learn to distinguish between correct word occurrences (positive examples) and incorrect ones (negative examples, often derived from ASR lattice confusions). This allows the models to focus on the confusions made by the baseline ASR system.
  • Novel Distance Metric: A key contribution is the introduction of the van Rossum distance metric. This metric, borrowed from computational neuroscience, measures the similarity between two point process patterns (sequences of events) by considering their temporal proximity, even if they aren't perfectly synchronized. This is an improvement over simpler methods like "binning," which discretizes time into fixed intervals.
  • Lattice Rescoring: The trained DPPMs are used to generate confidence scores for candidate word occurrences identified within ASR lattices. These scores are then used to rescore the lattice arcs, effectively improving the confidence measure for potential term detections.
  • Fusion: The paper also explores fusing the scores from the DPPMs with the original lattice posterior probabilities from the baseline ASR system, using linear regression.

Datasets

The experiments were conducted on the Augmented Multi-party Interaction (AMI) corpus.

  • The corpus consists of approximately 80 hours of meeting recordings from 118 meetings with 121 speakers.
  • The data was split into three folds for training, development, and evaluation, ensuring that no speaker overlap occurred between training and testing sets to avoid test-on-train violations.

Key Insights and Ideas

  • Point Processes as a Viable Alternative: The paper demonstrates that point process models, particularly DPPMs, can serve as a powerful alternative to traditional HMM-GMM acoustic models for spoken term detection, especially when applied to lattice rescoring.
  • The Power of the Van Rossum Distance: The van Rossum distance metric significantly outperforms simpler binning methods for comparing phonetic event patterns. This highlights the importance of using sophisticated temporal similarity measures when dealing with event-based representations of speech.
  • Complementarity of Different Pipelines: The DPPM-based approach, despite using a distinct pipeline (MLP-based phoneme detectors, point process models) and not employing techniques like VTLN or speaker adaptation, produces scores that are complementary to those from a high-performance HMM-GMM baseline (IBM Attila LVCSR). This suggests that combining information from different modeling paradigms can lead to substantial gains.
  • Fusion Yields Further Gains: Simply fusing the DPPM scores with the baseline lattice posteriors resulted in a notable improvement in performance, underscoring the value of the DPPMs as an additional source of information.

Key Results

  • The DPPMs, using the van Rossum distance, achieved an Equal Error Rate (EER) of 12.4%, which is a significant improvement over the binning method (14.0% EER).
  • This performance is close to that of the state-of-the-art IBM Attila LVCSR lattice posteriors, which achieved an EER of 7.8%.
  • When the DPPM scores were fused with the lattice posteriors, the EER was further reduced to 7.1%, demonstrating a 9.0% relative improvement over the baseline.
  • The Figure-of-Merit (FOM) also showed consistent improvements across the board, with the fused scores achieving the highest FOM of 91.6%.

Limitations and Future Work

  • No VTLN or Speaker Adaptation: The DPPM pipeline did not utilize VTLN or speaker adaptation techniques, which are known to improve ASR performance. The authors suggest that incorporating these could lead to even better results.
  • Limited to Lattice Arcs: The DPPMs only use acoustic information within the bounds of the given lattice arcs. Future work could explore extending this to utilize information beyond the immediate lattice arc.
  • Complementarity: The paper highlights the complementarity of DPPM scores with lattice posteriors. Future work could investigate more sophisticated fusion techniques beyond simple linear regression.
  • Word Set: The evaluation was performed on a selected set of 35 high-frequency words. Expanding the evaluation to a larger and more diverse set of words would be beneficial.

Summary updated

BibTeX
@inproceedings{norouzian2012exploiting,
  author = {Atta Norouzian and Aren Jansen and Richard C. Rose and Samuel Thomas},
  title = {{Exploiting discriminative point process models for spoken term detection}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure