On-the-fly lattice rescoring for real-time automatic speech recognition

Haşim Sak, Murat Saraçlar, Tunga GüngörInterspeech 2010

TL;DR

This paper introduces an "on-the-fly" lattice rescoring method for automatic speech recognition (ASR) systems. It allows for improved word accuracy by using a more powerful language model without introducing the latency typically associated with offline rescoring, making it suitable for real-time applications.

What Problem Does This Paper Address?

In large vocabulary Automatic Speech Recognition (ASR) systems, a trade-off exists between accuracy and computational cost. To achieve real-time performance, the initial decoding pass often uses a simplified, "pruned" language model. This results in a word lattice – a graph representing multiple possible transcriptions – that is then typically rescored offline with a more comprehensive language model to boost accuracy. However, this offline rescoring adds significant latency, making it unsuitable for real-time applications like automatic closed captioning. This paper tackles the challenge of improving ASR accuracy through lattice rescoring without sacrificing real-time performance.

Methods

The core innovation is an "on-the-fly" lattice rescoring algorithm. This method integrates lattice generation and rescoring directly into the decoding process.

  • WFST Framework: The paper leverages Weighted Finite-State Transducers (WFSTs) as a unified framework to represent various ASR knowledge sources (HMMs, lexicons, language models).
  • Lattice Generation: The lattice generation algorithm is based on Ljolje et al. [1]. It constructs a phone-to-word transducer lattice during the Viterbi decoding process. Each arc in this lattice carries acoustic and language model scores.
  • On-the-fly Rescoring: The key novelty lies in how the lattice is generated and rescored. Instead of generating a lattice with combined acoustic and language model scores, the proposed method generates a lattice containing only acoustic model scores. This allows for subsequent rescoring with a more powerful, unpruned language model. The rescoring happens concurrently with lattice generation, meaning the lattice is being built and rescored simultaneously as the speech is processed.
  • N-gram History Tracking: To handle the rescoring language model, each state in the rescored lattice keeps track of the n-gram history of the path leading to it. This allows the system to query the rescoring language model for the probability of the next word given the preceding words.
  • Implementation: The authors implemented a WFST-based Viterbi decoder using the OpenFst library and integrated their on-the-fly rescoring algorithm.

Datasets

  • Training: The acoustic model was trained on 188 hours of Turkish broadcast news speech data. Language models were trained on approximately 200 million words from the BOUN NewsCor corpus and 1.3 million words from the Turkish Broadcast News speech database.
  • Testing: The test set consisted of 3.1 hours of speech data, pre-segmented into 2,410 utterances.

Key Insights and Ideas

  • Decoupling Lattice Generation and Rescoring: By generating a lattice with only acoustic scores and then rescoring it, the system avoids the need to subtract initial language model scores, simplifying the process.
  • Real-time Rescoring is Possible: The primary insight is that lattice rescoring, traditionally an offline process, can be integrated into the real-time decoding loop without introducing prohibitive latency.
  • Flexibility in Rescoring Models: The framework is general enough to accommodate language models that cannot be represented as finite-state automata, such as discriminative language models.
  • Minimal Computational Overhead: The on-the-fly method achieves this accuracy improvement with less than 20% increased computation over standard lattice generation.

Key Results

  • The on-the-fly lattice rescoring method achieved comparable accuracy to offline rescoring methods.
  • Crucially, the on-the-fly method did not increase the latency of the system, maintaining a real-time factor of approximately 1 (CPU time / audio time) for the improved accuracy.
  • The proposed method demonstrated significant accuracy improvements for real-time systems compared to a baseline system with no rescoring.

Limitations and Future Work

  • N-gram History Management: While the paper uses n-gram histories, the authors note that the framework can be extended to more complex models, including discriminative models, which might require more sophisticated history management.
  • Parameter Estimation: The authors suggest that the decoder could be adapted for parameter estimation of discriminative language models using dynamically generated lattices, which is an interesting avenue for future research.
  • Multiple Rescoring Models: The framework is designed to allow for rescoring with multiple models, which could be explored further.

Summary updated

BibTeX
@inproceedings{sak2010fly,
  author = {Haşim Sak and Murat Saraçlar and Tunga Güngör},
  title = {{On-the-fly lattice rescoring for real-time automatic speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure