Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR

Minghan Wang, Yuxia Wang, Thuy-Trang Vu, Ehsan Shareghi, Gholamreza HaffariarXiv 2024 · DOI · arXiv

Abstract

Automatic Speech Recognition (ASR) systems often struggle with the specialized vocabulary and dense technical jargon found in scientific presentations. This paper introduces the Multimodal Scientific ASR (MS-ASR) task and proposes SciVASR, a framework that leverages visual information from presentation slides to correct technical errors in transcripts. We also introduce Severity-aware WER (SWER), a metric designed to better evaluate transcription quality in knowledge-intensive domains by weighting errors based on their semantic and technical significance. Our experiments on the ACL 60/60 dataset demonstrate that incorporating visual context leads to a 45% improvement in SWER compared to speech-only baselines.

The Challenge of Scientific ASR

Standard ASR models, typically trained on general-purpose speech datasets, frequently misinterpret niche technical terminology. For example, a model might transcribe the model name "BERT" as the common word "birds." This "jargon gap" is compounded by a metric problem: the standard Word Error Rate (WER) treats all errors equally. In a scientific context, however, misidentifying a core technical term is far more damaging to comprehension than a minor grammatical error or a missing filler word.

Comparison of Multimodal vs. Speech-Only ASR and the illustration of the Severity-aware WER (SWER) metric.

The SciVASR Framework and SWER Metric

To address these challenges, we present the SciVASR (Scientific Vision Augmented ASR) framework. Rather than training a new ASR model from scratch, SciVASR utilizes a post-editing pipeline that integrates visual context from presentation slides.

The pipeline consists of four main stages:

  1. Initial Transcription: Generating a baseline text using a robust ASR model like OpenAI’s Whisper.
  2. Video Segmentation: Identifying slide transitions to align visual content with the audio timeline.
  3. Visual Context Extraction: Using a Vision-LLM to extract key terms, titles, and bullet points directly from the slides.
  4. Post-Editing: A Text-LLM refines the initial transcript by cross-referencing it with the extracted visual "cheat sheet" to resolve technical inaccuracies.
The SciVASR pipeline: integrating ASR transcripts with visual context extracted from slides via Multimodal LLMs.

To evaluate this framework, we proposed Severity-aware WER (SWER). This metric categorizes errors into six types (e.g., Terminology, Numbers, Named Entities) and three severity levels (Critical, Minor, OK). By weighting critical technical errors five times more heavily than minor formatting issues, SWER provides a more accurate reflection of a transcript's utility for scientific readers.

Experimental Results and Insights

We evaluated SciVASR using the ACL 60/60 Dataset, which contains 10 high-density NLP conference presentations. Our key findings include:

  • Slides as Ground Truth: Presentation slides often contain the exact spelling of technical terms. Treating them as a "visual vocabulary" allows the model to resolve homophone errors that are impossible to distinguish via audio alone.
  • Significant Accuracy Gains: The full SciVASR pipeline using GPT-4o achieved a 45% improvement in SWER over speech-only baselines, specifically reducing terminology errors by over 150 instances in the test set.
  • Metric Validity: We found that standard WER can be misleading; some post-editing improvements significantly enhanced readability and technical accuracy while only marginally affecting (or even slightly increasing) the standard WER. SWER showed a 93% correlation with human expert evaluations.

Conclusion and Future Work

SciVASR demonstrates the power of Multimodal LLMs in bridging the jargon gap for scientific ASR. While the current multi-step pipeline is effective, future work will focus on reducing latency for real-time applications and expanding the dataset to other scientific domains like physics and biology to further validate the framework's versatility.

Summary updated

Abstract

Recent advancements in multimodal large language models (MLLMs) have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. This paper introduces the Multimodal Scientific ASR (MS-ASR) task, which focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. Realized that traditional metrics like WER fall short in assessing performance accurately, prompting the proposal of severity-aware WER (SWER) that considers the content type and severity of ASR errors. We propose the Scientific Vision Augmented ASR (SciVASR) framework as a baseline method, enabling MLLMs to improve transcript quality through post-editing. Evaluations of state-of-the-art MLLMs, including GPT-4o, show a 45% improvement over speech-only baselines, highlighting the importance of multimodal information integration.
BibTeX
@misc{wang2024exploring,
  author = {Minghan Wang and Yuxia Wang and Thuy-Trang Vu and Ehsan Shareghi and Gholamreza Haffari},
  title = {{Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR}},
  year = {2024},
  eprint = {2406.10880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.10880},
}
Figure