TL;DR
This paper tackles the challenge of combining multiple Automatic Speech Recognition (ASR) systems to improve accuracy without sacrificing real-time performance. They introduce a novel framework that integrates auxiliary ASR system hypotheses during the decoding process of a primary system, significantly reducing latency compared to traditional methods that combine outputs only at the end.
What Problem Does This Paper Address?
Modern Large Vocabulary Continuous Speech Recognition (LVCSR) systems achieve high accuracy by integrating various knowledge sources and often employing multi-pass decoding strategies. However, these sophisticated approaches demand significant computational resources and time, leading to increased latency. System combination, a technique that merges outputs from multiple ASR systems, further boosts accuracy but traditionally exacerbates the latency issue because it requires waiting for all individual systems to complete their full decoding passes. This paper addresses the critical need for ASR system combination that can deliver improved accuracy with minimal impact on real-time response, crucial for applications requiring low latency.
Methods
The core of the proposed solution lies in a parallelized, "on-the-fly" combination strategy. Instead of waiting for each ASR system to finish decoding an entire utterance, auxiliary systems provide their partial recognition hypotheses (auxiliary hypotheses) to the primary system during its decoding process. This allows for a more integrated and timely combination.
Two key mechanisms enable this framework:
- BONG driven decoding algorithm: This approach, an evolution of previous work, uses "bags of n-grams" (BONG) to represent auxiliary hypotheses. Instead of complex alignments, it involves a simpler search within these bags to integrate external information into the primary system's search algorithm. This makes it straightforward to add new auxiliary systems.
- Local ROVER combination: A modification of the classical ROVER (Recognizer Output Voting Error Reduction) algorithm is proposed. Instead of performing a pairwise alignment at the sentence level (which is time-consuming), this "local" version operates on smaller temporal segments. When the primary system emits a recognized word, this word and its recent history are compared with auxiliary hypotheses from the same temporal window. A voting mechanism based on word frequency is then applied locally.
Datasets
The experiments were conducted using a subset of the development data from the ESTER 2 evaluation campaign, which comprises approximately 6 hours of French radio broadcast news. The authors note that they used a wide-band subset of this data to ensure comparability of initial performance across systems.
Key Insights and Ideas
- Latency is a major bottleneck for system combination: The paper clearly articulates that traditional system combination methods, while accurate, are too slow for real-time applications due to their sequential, end-of-utterance processing.
- Parallelism is key to low-latency combination: By having auxiliary systems contribute during the primary system's decoding, the latency bottleneck is significantly reduced.
- BONG and Local ROVER are synergistic: Combining the BONG driven decoding (for integrating auxiliary hypotheses) with the Local ROVER (for timely, local voting) yields the best results, outperforming either method alone and classical ROVER.
- Trade-off management: The proposed framework demonstrates that it's possible to achieve a good speed/accuracy trade-off by parallelizing single-pass ASR systems and combining them on the fly.
Key Results
The paper reports significant improvements in Word Error Rate (WER) through their proposed framework:
- Individual System Performance: The baseline systems (LIUM, LIA, RASR, RASRLDA) had initial WERs ranging from 32.3% to 34.4%.
- BONG Combination: Using BONG to combine three auxiliary systems with the primary LIUM system resulted in a WER of 28.6%.
- Classical ROVER: Classical ROVER combination achieved a WER of 28.3%.
- BONG + Classical ROVER: Combining BONG with classical ROVER (but still waiting for full decoding) improved the WER to 27.4%.
- Local ROVER (LOROV): The Local ROVER combination alone achieved a WER of 30.8%, showing improvement over single systems but less than classical ROVER.
- Proposed Framework (LOROV + BONG): The combination of Local ROVER with BONG integration achieved a WER of 27.8%. This is a significant improvement over the best single system and is comparable to classical ROVER, but with the crucial advantage of reduced latency.
- Overall Improvement: The paper states that the proposed framework, when combining three state-of-the-art ASR systems, achieved a significant accuracy improvement of 4.5 absolute points in WER.
Limitations and Future Work
- Latency Impact: While the paper focuses on reducing latency due to system combination, it acknowledges that the speed of the ASR processing itself is not directly addressed. The latency reduction is relative to classical combination methods.
- Suboptimal Local Alignment: The authors note that the Local ROVER's accuracy is slightly suboptimal compared to classical ROVER due to its "local alignment" nature.
- Future Work: The authors express interest in increasing the number of auxiliary systems and leveraging their CORBA middleware implementation to allow recognizers to interoperate across platforms, further enhancing the scalability of their low-latency combination approach.
Summary updated