TL;DR
This paper introduces a novel approach to improve Automatic Speech Recognition (ASR) by leveraging imperfect textual transcripts (like subtitles or rough notes) that often accompany audio. By dynamically synchronizing these imperfect transcripts with the ASR's search process and adapting its language model, the system achieves significantly better recognition accuracy than using either the transcript alone or a standard ASR.
What Problem Does This Paper Address?
In many real-world scenarios, we have audio paired with textual information that should correspond to the spoken words, but it's rarely a perfect match. Think of movie subtitles, live captions, or even rough notes taken during a lecture. These "imperfect transcripts" can have significant word error rates (WER) compared to the actual speech. Traditional ASR systems struggle with this mismatch; alignment methods often fail, leading to large chunks of audio being ignored, and the ASR's performance suffers. The challenge is to harness the valuable, albeit imperfect, information within these transcripts to boost ASR accuracy, rather than being hindered by their inaccuracies.
Methods
The authors propose a two-pronged approach to integrate imperfect transcripts into the ASR decoding process, specifically within an A* search algorithm (used in their SPEERAL system):
- Language Model Adaptation: They create a language model specifically from the imperfect transcript. This model is then interpolated with a generic language model. This allows the system to benefit from the specific vocabulary and phrasing of the transcript while retaining robustness from the general language model, preventing overfitting to transcript errors.
- Dynamic Synchronization and Rescoring: This is the core innovation. Instead of a rigid alignment, they implement an "on-demand" synchronization. As the ASR's search progresses, it attempts to align the current best hypotheses with the imperfect transcript. If a match is found, the hypothesis's score is dynamically "rescored" to favor words that align with the transcript. This mechanism guides the search towards more likely paths based on the transcript, but crucially, it can switch back to a free-recognition mode if the acoustic evidence strongly contradicts the transcript, preventing the system from being led astray by transcript errors. This synchronization is achieved using Dynamic Time Warping (DTW) for efficient alignment.
Datasets
The experiments were conducted on a 3-hour subset of the ESTER evaluation campaign database, specifically using radio broadcast data from France Inter 1, France Inter 2, and France Info. The imperfect transcripts were created by manually introducing errors (10% and 20% WER) into the original, accurate transcripts to simulate real-world conditions.
Key Insights and Ideas
- Imperfect transcripts are valuable, but need careful handling: Simply using a language model trained on imperfect transcripts improves ASR, but the system can still be misled by transcript errors.
- Dynamic synchronization is key: The ability to align and rescore hypotheses during the search, rather than relying on a pre-computed alignment, is crucial for leveraging imperfect information without being penalized by its inaccuracies.
- Robustness through interpolation: Combining transcript-specific language models with generic ones prevents the ASR from becoming overly reliant on potentially flawed transcript data.
- Potential for speedup: While adding computational overhead, the dynamic synchronization can sometimes speed up decoding by guiding the search more efficiently in well-transcribed sections.
Key Results
The proposed transcript-driven system significantly outperformed the baseline ASR system.
- When using a language model interpolated with a 10% WER imperfect transcript, the Word Error Rate (WER) was reduced to 7.2% (compared to a baseline of 10.1% for the original transcript and higher for the baseline system).
- The best configuration, combining a generic language model (70%) with an imperfect transcript model (30%) and dynamic alignment, achieved a WER of 7.2% on one dataset and 7.7% on another, representing a relative WER improvement of 28% to 40% compared to the baseline.
- The performance gain was observed to be relatively independent of the initial transcript quality, showing robustness even with a 20% WER transcript.
Limitations and Future Work
The authors acknowledge that their experiments were conducted under controlled conditions (low noise, specific speech style and topics). They suggest future work should focus on applying these methods to more challenging scenarios, such as automatic subtitling of movies or theater plays, which often involve more diverse language, accents, and higher error rates in transcripts. They also note that search efficiency could be further improved by introducing more sophisticated heuristics in the probe function of the A* search.
Summary updated