Speech Recognition on TV Series with Video-guided Post-Correction

Haoyuan Yang, Yue Zhang, Liqiang Jing

Abstract

Automatic Speech Recognition (ASR) has achieved remarkable success with deep learning, driving advancements in conversational artificial intelligence, media transcription, and assistive technologies. However, ASR systems still struggle in complex environments such as TV series, where multiple speakers, overlapping speech, domain-specific terminology, and long-range contextual dependencies pose significant challenges to transcription accuracy. Existing approaches fail to explicitly leverage the rich temporal and contextual information available in the video. To address this limitation, we propose a Video-Guided Post-ASR Correction (VPC) framework that uses a Video-Large Multimodal Model (VLMM) to capture video context and refine ASR outputs. Evaluations on a TV-series benchmark show that our method consistently improves transcription accuracy in complex multimedia environments.

1. Introduction

Automatic Speech Recognition (ASR) has seen significant progress, yet "in-the-wild" scenarios like TV shows and movies remain a major challenge. These environments are characterized by overlapping speech, background noise, and the frequent use of domain-specific terminology or unique character names (e.g., "Joey Tribbiani"). Standard ASR models rely primarily on audio signals, leading to phonetic ambiguities and incorrect guesses in noisy conditions. While some research has explored using static images for grounding, these methods fail to capture the rich temporal context inherent in video. This paper introduces a framework that explicitly leverages video sequences to provide the "world knowledge" necessary to correct transcript errors post-generation.

2. Methodology

The proposed Video-Guided Post-ASR Correction (VPC) is a two-stage, training-free framework designed to refine ASR outputs using multimodal context.

2.1 ASR Generation

In the first stage, a standard pre-trained ASR model (such as WavLM or Conformer) generates an initial "raw" transcript from the audio signal.

2.2 Video-Guided Correction

The second stage refines the raw transcript through two sub-modules:

  • Contextual Information Extraction: The system employs VideoLLaMA2, a Video-Large Multimodal Model (VLMM), to analyze the video clip. It extracts descriptive text by identifying the TV show and providing a detailed caption of the actions and characters.
  • Context-Aware Correction: The raw transcript and the extracted visual descriptions are processed by GPT-4o. Acting as an editor, the LLM uses the visual context to identify and fix recognition errors—for instance, correcting "Joey Tribbyany" to "Joey Tribbiani" based on the identification of the show Friends.
The VPC framework utilizes VideoLLaMA2 for visual context extraction and GPT-4o for final transcript correction.

3. Experimental Setup

The framework was evaluated using the Violin-TV subset, curated from the larger Violin dataset. This subset consists of 10,003 clips (approximately 90 hours) from English-language TV series with high speech density. The data was split into 72 hours for training (base ASR fine-tuning), 9 hours for validation, and 9 hours for testing.

4. Results

The VPC framework consistently outperformed both raw ASR and text-only correction across various underlying models.

Quantitative results showing Word Error Rate (WER) improvements across different ASR models using the VPC framework.

Key performance highlights include:

  • WavLM-Large: The most significant improvement, with the Word Error Rate (WER) dropping from 29.83% to 23.64%, representing a 20.75% relative improvement.
  • wav2vec2-large: Achieved a relative improvement of 13.06%.
  • Conformer-large: Showed a 7.46% improvement.

The training-free nature of the framework ensures consistent gains regardless of the base ASR model used.

5. Key Insights

  • Visual Grounding: Visual context acts as a "spellcheck" anchor. For example, if a video depicts a beehive but the ASR output is "be hi hat," the LLM can leverage the visual description to select the correct term.
  • The Necessity of Multimodal Context: Correcting transcripts with GPT-4o without video context sometimes increased error rates. Without visual evidence, LLMs may "hallucinate" plausible but factually incorrect corrections.
  • Metadata Advantage: Identifying show titles allows the LLM to access internal knowledge about specific characters and settings, significantly improving the recognition of proper names.

6. Conclusion and Future Work

The VPC framework demonstrates the power of leveraging visual context for ASR correction in complex multimedia environments. However, challenges remain regarding the inference cost of large models like GPT-4o and VideoLLaMA2, as well as the dependency on the quality of VLMM-generated descriptions. Future research will explore deeper video understanding and more integrated multimodal strategies to incorporate visual signals earlier in the ASR pipeline.

Summary updated

Abstract

Automatic Speech Recognition (ASR) has achieved remarkable success with deep learning, driving advancements in conversational artificial intelligence, media transcription, and assistive technologies. However, ASR systems still struggle in complex environments such as TV series, where multiple speakers, overlapping speech, domain-specific terminology, and long-range contextual dependencies pose significant challenges to transcription accuracy. Existing approaches fail to explicitly leverage the rich temporal and contextual information available in the video. To address this limitation, we propose a Video-Guided Post-ASR Correction (VPC) framework that uses a Video-Large Multimodal Model (VLMM) to capture video context and refine ASR outputs. Evaluations on a TV-series benchmark show that our method consistently improves transcription accuracy in complex multimedia environments.
BibTeX
@misc{yang2025speech,
  author = {Haoyuan Yang and Yue Zhang and Liqiang Jing},
  title = {{Speech Recognition on TV Series with Video-guided Post-Correction}},
  year = {2025},
  eprint = {2506.07323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.07323},
}
Figure