TL;DR
This paper introduces SlideAVSR, a new dataset of scientific paper explanation videos designed to push audio-visual speech recognition (AVSR) beyond simple lip-reading tasks. They also propose DocWhisper, an AVSR model that leverages optical character recognition (OCR) of on-screen slides to significantly improve transcription accuracy, especially for technical terms.
What Problem Does This Paper Address?
Audio-visual speech recognition (AVSR) aims to improve speech recognition by incorporating visual cues, primarily lip movements. While AVSR models have shown promise on lip-reading datasets, their effectiveness in real-world scenarios, especially those involving complex technical language and diverse visual contexts, remains limited. Existing AVSR datasets often fall short in evaluating an AVSR model's ability to comprehend visual information beyond just facial features. This paper addresses the challenge of transcribing technical jargon in paper explanation videos, where the spoken words are often difficult to discern without referring to the accompanying slides.
Methods
The core innovation lies in the SlideAVSR dataset construction and the DocWhisper model.
For the dataset, the authors employed a multi-stage filtering process:
- Data Collection: Videos were collected using search queries related to top AI conferences from 2020-2023.
- Filtering: A combination of automated filters (ChatGPT for video topic verification, BLIP-2 for slide verification) and manual checks were used to ensure videos were indeed paper explanations and contained slides.
- Cleansing: This involved audio-subtitle alignment, removing utterances not corresponding to slides, and merging short utterances.
- Data Partitioning: The dataset was partitioned into Train, Dev, and Test sets, with a specific TestB subset for South Asian English (SAE) accents to evaluate accent robustness.
The DocWhisper model enhances a standard AVSR model (Whisper) by integrating OCR.
- OCR Integration: The model uses OCR to extract text from the slides shown in the video.
- Prompting: This extracted text is then used as a prompt for the Whisper model during fine-tuning and inference. This differs from previous work that used CLIP-derived prompts for zero-shot learning.
- FQ Ranker: To address the long-tail nature of OCR results and prioritize important terms, they introduce FQ Ranker, which ranks words based on their frequency. This aims to improve the information content of the prompts.
Datasets
- SlideAVSR (New Dataset): This is the primary contribution, consisting of approximately 36 hours of audio-visual data from 245 scientific paper explanation videos. It's designed to challenge AVSR models with technical terminology and diverse visual content.
- Whisper (Base Model): Used as the foundation for their AVSR model.
- English Wikipedia: Used for obtaining word frequency counts for the FQ Ranker.
Key Insights and Ideas
- Slides are Crucial for Technical AVSR: The paper strongly suggests that for AVSR tasks involving technical content, leveraging on-screen text (like slides) is not just beneficial but essential for accurate transcription.
- Beyond Lip-Reading: The dataset highlights the need for AVSR research to move beyond purely lip-reading tasks and consider broader visual comprehension, including text on screen.
- OCR as a Powerful Prompt: Using OCR-generated text from slides as prompts for AVSR models is a simple yet highly effective way to boost performance, especially for challenging vocabulary.
- FQ Ranker's Nuance: While FQ Ranker shows promise, its effectiveness is sensitive to the prompt length. It seems to help more with shorter prompts, suggesting that longer prompts might disrupt the contextual flow for the AVSR model's decoder.
- Accent Robustness is Still a Challenge: Even with visual cues, AVSR models struggle with less common accents, underscoring the ongoing need for diverse training data.
Key Results
- DocWhisper Performance Boost: DocWhisper achieved a maximum improvement of 14.3% on the TestA set and 11% on the TestB set compared to a fine-tuned Whisper model.
- FQ Ranker Impact: FQ Ranker improved scores on TestB for shorter prompt lengths (25 words), but performance degraded with longer prompts (over 50 words).
- Error Correction: DocWhisper was most effective at correcting substitution errors in Whisper. These corrections included technical terms (41%), inflections (28%), and mishearings (24%).
- Dataset Size: The SlideAVSR dataset contains approximately 36 hours of data from 245 videos and 19,463 utterances.
Limitations and Future Work
- Dataset Size: SlideAVSR has a relatively limited number of videos and speakers compared to mainstream AVSR datasets, which could lead to data imbalance.
- Bias in Data: The focused collection on AI paper explanations might introduce imbalances in terms of speaker nationality, age, and gender.
- Manual Annotation Reliance: While they used vision-language models to reduce annotation costs, the dataset construction still relies on manual annotation.
- Accent Classification Difficulty: Accurately classifying accents was challenging, leading to a selective focus on SAE accents for the TestB set.
Future work includes:
- Refining OCR-based methods and developing end-to-end AVSR models independent of OCR.
- Creating a benchmark that comprehensively evaluates AVSR models across diverse video types (e.g., sports, gaming, cooking).
- Building a foundation model for AVSR with high performance across various video inputs.
- Fully automating the dataset construction process.
Summary updated