EP-LVMQA: Evidence-Preserving Coarse-to-Fine Localization for Low-Quality Long Video Moment Question Answering

Guoning Chen, Zhexun Hu, Yifan Wei, Yixuan Weng, Bin Li, Yufei ZhaoarXiv 2026 · arXiv

TL;DR

EP-LVMQA introduces a framework for long-video question answering that prioritizes inspectable evidence over just returning text or labels. By utilizing a novel "evidence card" to preserve multimodal metadata (like source confidence and matched cues) between planning and localization stages, the system achieves state-of-the-art results on medical instructional videos, even when data is noisy, sparse, or conflicting.

What Problem Does This Paper Address?

Instructional videos—think medical tutorials or industrial training—are often long and complex. When a user asks a question like "How should the bandage be layered?", they don't just need a text answer; they need to see the specific moment in the video that proves it.

Current Video Question Answering (VideoQA) systems face two major hurdles:

  1. Low-Quality Data: Real-world videos often have noisy Automated Speech Recognition (ASR), sparse Optical Character Recognition (OCR), or visual actions that occur without any narration.
  2. The "Black Box" Handoff: Most "coarse-to-fine" systems retrieve a candidate clip and then pass only the raw features to a localizer. This discards the "provenance"—the information about why that clip was chosen, which modality (text vs. vision) was most confident, and what specific cues were matched.

The authors argue that localization quality suffers when this metadata is lost, leading to loose boundaries and results that are difficult for humans to verify.

Methods

The authors propose EP-LVMQA (Evidence-Preserving Coarse-to-Fine Localization), which consists of three main modules:

  • Uncertainty-Aware Multimodal Evidence Timeline: This module aligns visual features, ASR, OCR, and generated captions into overlapping windows. Crucially, it calculates a "confidence proxy" for each source, helping the model decide which modality to trust when data is missing or noisy.
  • Coarse Evidence Planner: Instead of searching the whole video at once, the planner uses "Question-Conditioned Evidence Cues" (extracted actions, objects, and domain-specific terms) to select the top $K$ candidate windows.
  • Evidence-Conditioned Fine Localizer: This is where the magic happens. The planner passes an Evidence Card to the localizer. This card isn't just a clip; it contains planner scores, source-specific payloads, availability masks, and a "temporal prior" (a Gaussian distribution centered on the most likely moment).
Figure 1: (a) Text-driven methods pass ASR/OCR-retrieved clips or scores to a clip-conditioned localizer, yielding a loose interval without provenance. (b) EP-LVMQA transfers multimodal evidence and source-aware metadata in an evidence card, enabling routed, prior-guided refinement and an inspectable primary evidence moment.

The localizer uses a transformer to fuse these routed tokens, applying the planner's prior as a "soft boundary bias" to sharpen the final start and end timestamps. The final output includes an Evidence Trace, allowing users to see exactly which cues and modalities led to the result.

Datasets

  • MedVidQA: The primary benchmark, consisting of medical instructional videos where questions require precise temporal grounding.
  • Ego4D-NLQ & QVHighlights: Used as cross-domain benchmarks to ensure the method generalizes beyond medical videos.
  • Human-Audit Set: The authors curated a 300-query subset for human evaluation, focusing on "Sufficiency" (does the clip answer the question?) and "Verifiability" (is the evidence clearly identifiable and well-scoped?).

Key Insights and Ideas

  • Provenance as a First-Class Citizen: The core "aha moment" is that preserving the reasoning chain (the evidence card) actually improves the final localization accuracy. It's not just for human readability; it's a better way to train the model.
  • Handling Modal Conflict: In instructional videos, the narrator might describe a step after they've performed it, or the ASR might be garbled. By using source-aware routing, EP-LVMQA can "ignore" a noisy ASR stream if the visual evidence is stronger.
  • Boundary Sharpening: The use of a temporal prior from the planner acts as a powerful guide for the localizer, leading to much tighter boundaries than standard methods.
Figure 2: EP-LVMQA comprises three modules: (a) an Uncertainty-Aware Multimodal Evidence Timeline, (b) a Coarse Evidence Planner, and (c) an Evidence-Conditioned Fine Localizer. Question-Conditioned Evidence Cues condition the planner and localizer, which communicate through an Evidence-Preserving Candidate Interface implemented as an evidence card.

Key Results

  • MedVidQA Performance: EP-LVMQA achieved 26.1 [email protected] and 31.8 mIoU, outperforming the strongest baselines by 6.3 and 5.1 points, respectively.
  • Human Utility: The gains were even more pronounced in human audits. "Sufficiency" and "Verifiability" scores improved by nearly 60% compared to non-evidence-preserving methods.
  • Robustness: When the authors intentionally degraded the input (e.g., removing 50% of ASR data or introducing conflicting captions), EP-LVMQA showed significantly lower performance drops (11.1% vs. 14.6% for the best baseline).
Figure 4: Frame-level comparisons under complete, miss- ing, and conflicting evidence. EP-LVMQA is tighter or more resilient than TimeZero, retaining target overlap with incom- plete or inconsistent evidence.

Limitations and Future Work

  • Absolute Performance: While a significant step forward, the absolute "Sufficiency" score remains at 25.4%. This suggests that long-video QA is still far from being reliable enough for autonomous medical or industrial use.
  • Causal Faithfulness: The authors note that while the "evidence trace" supports inspection, it doesn't strictly prove that the model's internal weights relied on those specific cues in a causal way.
  • Planner Dependency: The system is a pipeline; if the coarse planner fails to include the correct window in its top $K$ candidates, the fine localizer has no chance of recovering the correct answer. Future work might explore tighter end-to-end integration.

Summary updated

Abstract

Long instructional videos are increasingly used in medical education, industrial training, and procedural knowledge transfer, yet their evidence is often incomplete or unreliable: subtitles may be missing, ASR noisy, OCR sparse, and decisive actions briefly shown without narration. Existing VideoQA systems typically return labels or free-form text, making answer support difficult to inspect. We study Long Video Moment Question Answering (LVMQA), where a system returns timestamped answer-supporting evidence. We propose EP-LVMQA, an evidence-preserving coarse-to-fine framework comprising an Uncertainty-Aware Multimodal Evidence Timeline, a Coarse Evidence Planner, and an Evidence-Conditioned Fine Localizer. An evidence card transfers planner scores, matched cues, source-confidence proxies, modality-specific evidence, neighboring context, and a temporal prior between the latter two modules. On MedVidQA, EP-LVMQA achieves 26.1 [email protected] and 31.8 mIoU, improving the strongest non-ours results by 6.3 and 5.1 points. It raises Suff.@2 and Ver.@2 to 25.4 and 22.6, gains of 9.5 and 8.8 points. Under matched low-quality perturbations, average relative degradation is reduced from the strongest non-ours value of 14.6% to 11.1%. On Ego4D-NLQ and QVHighlights, EP-LVMQA remains broadly comparable to established task-specific localization methods.
BibTeX
@misc{chen2026ep,
  author = {Guoning Chen and Zhexun Hu and Yifan Wei and Yixuan Weng and Bin Li and Yufei Zhao},
  title = {{EP-LVMQA: Evidence-Preserving Coarse-to-Fine Localization for Low-Quality Long Video Moment Question Answering}},
  year = {2026},
  eprint = {2611.04913},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04913},
}
Figure