Abstract
Automatic Speech Recognition (ASR) has achieved remarkable success with deep learning, driving advancements in conversational artificial intelligence, media transcription, and assistive technologies. However, ASR systems still struggle in complex environments such as TV series, where multiple speakers, overlapping speech, domain-specific terminology, and long-range contextual dependencies pose significant challenges to transcription accuracy. Existing approaches fail to explicitly leverage the rich temporal and contextual information available in the video. To address this limitation, we propose a Video-Guided Post-ASR Correction (VPC) framework that uses a Video-Large Multimodal Model (VLMM) to capture video context and refine ASR outputs. Evaluations on a TV-series benchmark show that our method consistently improves transcription accuracy in complex multimedia environments.
1. Introduction
Automatic Speech Recognition (ASR) has seen significant progress, yet "in-the-wild" scenarios like TV shows and movies remain a major challenge. These environments are characterized by overlapping speech, background noise, and the frequent use of domain-specific terminology or unique character names (e.g., "Joey Tribbiani"). Standard ASR models rely primarily on audio signals, leading to phonetic ambiguities and incorrect guesses in noisy conditions. While some research has explored using static images for grounding, these methods fail to capture the rich temporal context inherent in video. This paper introduces a framework that explicitly leverages video sequences to provide the "world knowledge" necessary to correct transcript errors post-generation.
2. Methodology
The proposed Video-Guided Post-ASR Correction (VPC) is a two-stage, training-free framework designed to refine ASR outputs using multimodal context.
2.1 ASR Generation
In the first stage, a standard pre-trained ASR model (such as WavLM or Conformer) generates an initial "raw" transcript from the audio signal.
2.2 Video-Guided Correction
The second stage refines the raw transcript through two sub-modules:
- Contextual Information Extraction: The system employs VideoLLaMA2, a Video-Large Multimodal Model (VLMM), to analyze the video clip. It extracts descriptive text by identifying the TV show and providing a detailed caption of the actions and characters.
- Context-Aware Correction: The raw transcript and the extracted visual descriptions are processed by GPT-4o. Acting as an editor, the LLM uses the visual context to identify and fix recognition errors—for instance, correcting "Joey Tribbyany" to "Joey Tribbiani" based on the identification of the show Friends.
3. Experimental Setup
The framework was evaluated using the Violin-TV subset, curated from the larger Violin dataset. This subset consists of 10,003 clips (approximately 90 hours) from English-language TV series with high speech density. The data was split into 72 hours for training (base ASR fine-tuning), 9 hours for validation, and 9 hours for testing.
4. Results
The VPC framework consistently outperformed both raw ASR and text-only correction across various underlying models.
Key performance highlights include:
- WavLM-Large: The most significant improvement, with the Word Error Rate (WER) dropping from 29.83% to 23.64%, representing a 20.75% relative improvement.
- wav2vec2-large: Achieved a relative improvement of 13.06%.
- Conformer-large: Showed a 7.46% improvement.
The training-free nature of the framework ensures consistent gains regardless of the base ASR model used.
5. Key Insights
- Visual Grounding: Visual context acts as a "spellcheck" anchor. For example, if a video depicts a beehive but the ASR output is "be hi hat," the LLM can leverage the visual description to select the correct term.
- The Necessity of Multimodal Context: Correcting transcripts with GPT-4o without video context sometimes increased error rates. Without visual evidence, LLMs may "hallucinate" plausible but factually incorrect corrections.
- Metadata Advantage: Identifying show titles allows the LLM to access internal knowledge about specific characters and settings, significantly improving the recognition of proper names.
6. Conclusion and Future Work
The VPC framework demonstrates the power of leveraging visual context for ASR correction in complex multimedia environments. However, challenges remain regarding the inference cost of large models like GPT-4o and VideoLLaMA2, as well as the dependency on the quality of VLMM-generated descriptions. Future research will explore deeper video understanding and more integrated multimodal strategies to incorporate visual signals earlier in the ASR pipeline.
Summary updated