TL;DR
EP-LVMQA introduces a framework for long-video question answering that prioritizes inspectable evidence over just returning text or labels. By utilizing a novel "evidence card" to preserve multimodal metadata (like source confidence and matched cues) between planning and localization stages, the system achieves state-of-the-art results on medical instructional videos, even when data is noisy, sparse, or conflicting.
What Problem Does This Paper Address?
Instructional videos—think medical tutorials or industrial training—are often long and complex. When a user asks a question like "How should the bandage be layered?", they don't just need a text answer; they need to see the specific moment in the video that proves it.
Current Video Question Answering (VideoQA) systems face two major hurdles:
- Low-Quality Data: Real-world videos often have noisy Automated Speech Recognition (ASR), sparse Optical Character Recognition (OCR), or visual actions that occur without any narration.
- The "Black Box" Handoff: Most "coarse-to-fine" systems retrieve a candidate clip and then pass only the raw features to a localizer. This discards the "provenance"—the information about why that clip was chosen, which modality (text vs. vision) was most confident, and what specific cues were matched.
The authors argue that localization quality suffers when this metadata is lost, leading to loose boundaries and results that are difficult for humans to verify.
Methods
The authors propose EP-LVMQA (Evidence-Preserving Coarse-to-Fine Localization), which consists of three main modules:
- Uncertainty-Aware Multimodal Evidence Timeline: This module aligns visual features, ASR, OCR, and generated captions into overlapping windows. Crucially, it calculates a "confidence proxy" for each source, helping the model decide which modality to trust when data is missing or noisy.
- Coarse Evidence Planner: Instead of searching the whole video at once, the planner uses "Question-Conditioned Evidence Cues" (extracted actions, objects, and domain-specific terms) to select the top $K$ candidate windows.
- Evidence-Conditioned Fine Localizer: This is where the magic happens. The planner passes an Evidence Card to the localizer. This card isn't just a clip; it contains planner scores, source-specific payloads, availability masks, and a "temporal prior" (a Gaussian distribution centered on the most likely moment).
The localizer uses a transformer to fuse these routed tokens, applying the planner's prior as a "soft boundary bias" to sharpen the final start and end timestamps. The final output includes an Evidence Trace, allowing users to see exactly which cues and modalities led to the result.
Datasets
- MedVidQA: The primary benchmark, consisting of medical instructional videos where questions require precise temporal grounding.
- Ego4D-NLQ & QVHighlights: Used as cross-domain benchmarks to ensure the method generalizes beyond medical videos.
- Human-Audit Set: The authors curated a 300-query subset for human evaluation, focusing on "Sufficiency" (does the clip answer the question?) and "Verifiability" (is the evidence clearly identifiable and well-scoped?).
Key Insights and Ideas
- Provenance as a First-Class Citizen: The core "aha moment" is that preserving the reasoning chain (the evidence card) actually improves the final localization accuracy. It's not just for human readability; it's a better way to train the model.
- Handling Modal Conflict: In instructional videos, the narrator might describe a step after they've performed it, or the ASR might be garbled. By using source-aware routing, EP-LVMQA can "ignore" a noisy ASR stream if the visual evidence is stronger.
- Boundary Sharpening: The use of a temporal prior from the planner acts as a powerful guide for the localizer, leading to much tighter boundaries than standard methods.
Key Results
- MedVidQA Performance: EP-LVMQA achieved 26.1 [email protected] and 31.8 mIoU, outperforming the strongest baselines by 6.3 and 5.1 points, respectively.
- Human Utility: The gains were even more pronounced in human audits. "Sufficiency" and "Verifiability" scores improved by nearly 60% compared to non-evidence-preserving methods.
- Robustness: When the authors intentionally degraded the input (e.g., removing 50% of ASR data or introducing conflicting captions), EP-LVMQA showed significantly lower performance drops (11.1% vs. 14.6% for the best baseline).
Limitations and Future Work
- Absolute Performance: While a significant step forward, the absolute "Sufficiency" score remains at 25.4%. This suggests that long-video QA is still far from being reliable enough for autonomous medical or industrial use.
- Causal Faithfulness: The authors note that while the "evidence trace" supports inspection, it doesn't strictly prove that the model's internal weights relied on those specific cues in a causal way.
- Planner Dependency: The system is a pipeline; if the coarse planner fails to include the correct window in its top $K$ candidates, the fine localizer has no chance of recovering the correct answer. Future work might explore tighter end-to-end integration.
Summary updated