Combining multiple speech recognizers using voting and language model information

Holger Schwenk, Jean-Luc GauvainInterspeech 2000 · DOI

TL;DR

This paper explores and enhances the ROVER (Recognizer Output Voting Error Reduction) technique for combining multiple speech recognition systems. By introducing novel methods for handling ties, normalizing system outputs, and crucially, incorporating language model information, the authors achieve significant reductions in word error rates, demonstrating a more robust and effective approach to multi-system speech recognition.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems, while impressive, still make errors. A common strategy to improve accuracy is to use multiple ASR systems and combine their outputs. The ROVER algorithm, introduced by NIST, is a popular method for this, achieving notable improvements by "voting" on the most likely word at each point in the transcription. However, the original ROVER algorithm has limitations: it can be sensitive to the order in which systems are combined, it doesn't explicitly handle variations in system output quality, and it ignores valuable language model (LM) information that could help resolve ambiguities. This paper aims to address these shortcomings by proposing and evaluating several extensions to the ROVER framework.

Methods

The authors reimplemented the ROVER algorithm and introduced several key modifications:

  • Order of Combination: Investigated the impact of combining systems in different orders, particularly focusing on using the best-performing systems first.
  • Normalization/Filtering: Explored applying pre-processing steps to normalize and filter the outputs of individual recognizers before combining them. This aims to standardize the input to the ROVER process.
  • Treatment of Ties: Addressed the issue where multiple words might receive the same "vote" count. The original ROVER breaks these ties arbitrarily. The authors propose using language model (LM) information to make more informed decisions during tie-breaking.
  • Incorporation of Language Model Information: This is a central contribution. Instead of just relying on word counts, the authors use the LIMSI Broadcast News system's LM to select the word sequence that minimizes perplexity among the alternatives, especially when ties occur.

Datasets

The primary dataset used for evaluation is the Broadcast News evaluation set from 1998 and 1999. These datasets are standard benchmarks for evaluating large vocabulary continuous speech recognition (LVCSR) systems. The paper refers to results from NIST evaluations, indicating the use of official test sets.

Key Insights and Ideas

  • Not All Systems Are Created Equal: Combining too many systems, especially those with high error rates, can actually hurt performance. There's an optimal number of systems to combine, and it's not necessarily "more is better."
  • Order Matters (But Not Always Dramatically): While the order of combining systems can affect results, the impact is often less significant than other factors, especially when dealing with many systems.
  • Language Models are Crucial for Ambiguity: The most significant finding is that incorporating language model information to break ties dramatically improves performance. This suggests that even though ROVER works without explicit LM integration, it leaves substantial room for improvement by leveraging contextual information.
  • The "Oracle" is Surprisingly Good: The authors show that if they could always pick the correct word (an "oracle" scenario), the error rate would be very low. This indicates that the raw outputs from multiple recognizers contain a lot of correct information, and better combination strategies can unlock this potential.

Key Results

  • The original ROVER algorithm reduced word error rates by up to 20% on the 1998 Broadcast News evaluation.
  • The proposed modifications, particularly the use of LM information for tie-breaking, achieved an additional 5% relative word error reduction on the 1998 and 1999 Broadcast News test sets.
  • When using LM to break ties, combining just two systems yielded an 8.1% relative word error reduction compared to the best individual system.
  • For the 1999 Broadcast News evaluation, combining three relatively fast (10xRT) recognizers with LM tie-breaking achieved a word error rate of 13.6%, a significant improvement over the alphabetical ROVER baseline.

Limitations and Future Work

  • Normalization Completeness: The authors suggest that applying all possible filtering and normalization rules might yield further improvements, but this was not fully implemented.
  • Confidence Scores: While confidence scores from individual systems could help break ties, only a few participants provided them, and their comparability was an issue.
  • Computational Cost: While the reimplemented ROVER is efficient (0.01xRT on a SGI UNIX workstation), research on alignment procedures supporting variants or n-best lists as input is ongoing.
  • Future Directions: The paper highlights the potential for further improvements by exploring more sophisticated alignment procedures and potentially integrating LM information more deeply into the alignment process itself. The success of combining fast recognizers also suggests a promising direction for future research.

Summary updated

BibTeX
@inproceedings{schwenk2000combining,
  author = {Holger Schwenk and Jean-Luc Gauvain},
  title = {{Combining multiple speech recognizers using voting and language model information}},
  booktitle = {Interspeech},
  year = {2000},
  doi = {10.21437/icslp.2000-419},
}
Figure