What You See is What You Ask: Evaluating Audio Descriptions

Divy Kala, Eshika Khandelwal, Makarand TapaswiEMNLP 2025

TL;DR

This paper introduces ADQA, a novel benchmark for evaluating audio descriptions (ADs) that moves beyond short, isolated clips to assess ADs' effectiveness in helping visually impaired users understand movie narratives and appreciate visual details. By analyzing the inherent subjectivity of ADs and proposing a question-answering framework, ADQA reveals a significant gap between current automatic AD generation methods and human-authored descriptions.

What Problem Does This Paper Address?

Audio descriptions (ADs) are crucial for making visual media accessible to Blind and Low Vision (BLV) individuals. However, existing methods for automatically generating ADs primarily focus on evaluating them against single ground-truth references for very short, trimmed video clips. This approach fails to capture the inherent subjectivity in AD creation (different experts may describe the same scene differently) and doesn't assess whether the generated ADs truly help users understand the broader narrative or appreciate visual richness over longer segments. The paper aims to bridge this gap by proposing a more holistic evaluation framework.

Methods

The core of the paper's contribution is the ADQA (Audio Description Question Answering) benchmark. This framework evaluates ADs by treating them as context for a large language model (LLM) to answer questions.

Key methodological aspects include:

  • Video Segments over Trimmed Clips: Instead of short clips, ADQA uses longer video segments (typically 2-3 minutes) to provide more narrative context.
  • Question Generation: Two types of multiple-choice questions are generated:
  • Visual Appreciation (VA) questions: Based on factual details from ADs, testing if the AD highlights key visual elements.
  • Narrative Understanding (NU) questions: Based on plot summaries or descriptions, testing if the AD helps users follow the story.
  • LLM-based QA: An LLM (Gemini 2.5 Pro for generation, Gemini 2.0 Flash for answering) is used to generate questions and answer them based on provided context (dialogue + ADs).
  • Rationale-based Evaluation: The LLM is prompted to provide rationales for its answers, allowing for verification of whether the answer was derived from the provided context or prior knowledge. This is crucial for assessing the AD's utility.
  • AD Alignment: To quantify subjectivity, the authors developed a method to align multiple human-authored AD tracks for the same movie, revealing significant differences in content and emphasis.

Datasets

  • CMD-AD: Used for creating the ADQA benchmark. Videos are sourced from a YouTube channel featuring movie plot points.
  • MAD-eval: Also used for ADQA. ADs are sourced from AudioVault and LSMDC.
  • AudioVault: A non-profit repository of movie ADs used for generating ADs and as ground truth.
  • LSMDC: Used for training and evaluation in prior work, and as a source for ADs in MAD-eval.

The paper introduces ADQA as a new benchmark, differing from existing ones by focusing on longer video segments and a QA-based evaluation that directly assesses the utility of ADs for understanding and appreciation.

Key Insights and Ideas

  • Subjectivity is Pervasive: Analyzing multiple AD tracks for the same movie reveals that experts often choose to describe different visual details or highlight them with different wording, challenging the notion of a single "correct" AD.
  • Trimmed Clips are Insufficient: Evaluating ADs on short, isolated clips doesn't reflect how they function in a real viewing experience, where context and narrative flow are essential.
  • ADQA Measures True Utility: The QA framework directly tests whether ADs help users understand the story and appreciate visuals, going beyond simple caption similarity.
  • LLMs Can Generate and Answer QA: The study demonstrates the capability of LLMs to generate relevant questions and answer them based on ADs, while also highlighting the need to ensure answers are context-grounded.
  • Human ADs are Still the Gold Standard: Even advanced models struggle to match the quality and utility of human-authored ADs, particularly for narrative understanding.

Key Results

  • ADQA Benchmark: The paper presents ADQA with a substantial number of video segments, VA questions, and NU questions.
  • Subjectivity Quantification: Analysis of aligned AD tracks showed that even with high temporal overlap, ADs often differ significantly in content and wording, with 25-30% of ADs remaining non-aligned.
  • LLM Performance: LLMs can answer questions with no context, but their ability to ground answers in provided ADs is crucial for evaluating AD quality. ADs significantly improve narrative understanding and visual appreciation when combined with dialogue.
  • Model Evaluation: Human-authored ADs consistently outperform automatic generation methods on ADQA. For example, on CMD-AD NU, human ADs achieve CC scores around 73%, while top models like AutoAD-III reach only 63.5%. For VA, the gap is even larger.
  • Accuracy Ratio: This metric, measuring the gap closed between human performance and dialog-only context, highlights the significant room for improvement in current AD generation methods.

Limitations and Future Work

  • Temporal Feasibility: The current framework does not evaluate whether generated ADs fit naturally within dialogue gaps, which could lead to models saturating silent periods with verbose descriptions.
  • LLM Output Variability: LLM outputs can be non-deterministic, raising reproducibility concerns. The authors plan to mitigate this with a server-based evaluation.
  • Prior Knowledge Leakage: LLMs may leverage prior knowledge of popular movies, potentially inflating scores. However, since the same model is used for all evaluations, relative scores remain valid.
  • Future Work:
    • Video-level Evaluation: Moving beyond clips to full video segments is essential.
  • Focus on Narratives: Developing models that better connect events into coherent stories.
  • Leveraging VLMs: Exploring Vision-Language Models for extracting rich details for concise ADs.
  • Script Integration: Investigating how to effectively use movie scripts, acknowledging their potential unavailability.
  • Holistic Evaluation: Moving beyond LLM-based QA to more comprehensive evaluations that align with AD guidelines.

Summary updated

BibTeX
@inproceedings{kala2025what,
  author = {Divy Kala and Eshika Khandelwal and Makarand Tapaswi},
  title = {{What You See is What You Ask: Evaluating Audio Descriptions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure