TL;DR
EARS (Evidence-Aware Retrieval via Sound and motion) is a training-free, query-free front-end that selects the most informative frames for video reasoning by identifying spikes in motion and audio. It significantly outperforms uniform sampling for sparse-event videos under tight frame budgets, providing a "cheap default" for Vision-Language Models (VLMs) without the overhead of learned selectors or extra model passes.
What Problem Does This Paper Address?
Modern Vision-Language Models (VLMs) are powerful but computationally expensive, especially for long videos. To stay within "context budgets," researchers typically sample a fixed number of frames ($K$) uniformly across time. However, in many real-world scenarios like surveillance or safety monitoring, critical events are brief and sparse. Uniform sampling often misses these "needles in the haystack" entirely.
While "smart" frame selectors exist, they usually require a specific user query (e.g., "find the person fighting"), additional training, or expensive auxiliary VLM passes to rank frames. The authors identify a need for a lightweight, "evidence-centric" approach that can prioritize information-rich regions before any heavy VLM computation begins, even when no specific query is available.
Methods
The authors propose EARS, a three-stage pipeline that operates as a "pre-filter" for any downstream VLM:
- Multimodal Signal Extraction: It extracts two "cheap" signals: motion magnitude (using classical Farneback optical flow) and audio energy (short-time energy of the waveform).
- Evidence Scoring: These signals are normalized per-video and fused into a single scalar "evidence score" ($A_t$). This score acts as a proxy for information density—where there is high motion or loud sound, there is likely something worth looking at.
- Information Compression (IC): To avoid wasting the frame budget on redundant frames from the same event (e.g., 10 frames of the same punch), EARS applies a temporal Non-Maximum Suppression (NMS) heuristic. It greedily picks the highest-scoring frame and suppresses its neighbors within a minimum temporal gap ($\tau$), forcing the selection to be diverse.
Datasets
The method was evaluated on three distinct types of video benchmarks:
- XD-Violence: A large-scale surveillance dataset where anomalies (fights, explosions) typically have strong audio-visual signatures.
- UCF-Crime: A surveillance dataset featuring mostly silent or visually subtle anomalies (like shoplifting).
- EgoSchema: A dense-narrative benchmark consisting of long egocentric videos where every second contributes to a complex plot.
Key Insights and Ideas
- Evidence vs. Events: EARS doesn't try to classify what is happening (which is the VLM's job); it simply identifies where something information-bearing is happening.
- The "Small-K" Advantage: The gains from EARS are most dramatic when the frame budget is extremely tight ($K \le 4$). In this regime, picking the right frame is far more important than seeing more frames.
- Regime-Conditional Utility: EARS is not a universal "drop-in" replacement for uniform sampling. It is a specialized tool for sparse, loud, or fast events. On dense narratives (EgoSchema) or silent, subtle crimes (UCF-Crime), uniform sampling remains superior.
- The Routing Paradox: The authors found that while an "oracle" could perfectly choose when to use EARS vs. Uniform to get a massive performance boost, building a simple automated "router" to make that choice per-video is surprisingly difficult. Cheap statistics alone cannot reliably predict if a video requires semantic (VLM-level) attention.
Key Results
- XD-Violence Performance: On the XD-Violence dataset at $K=1$, EARS achieved an F1 score of 0.660, significantly beating the centered Uniform baseline of 0.614. At $K=2$, the gap remained substantial (0.817 vs 0.791).
- Localization Accuracy: When used to select frames for a VLM, EARS reached a frame-level localization Average Precision (AP) of 0.944, compared to 0.881 for uniform sampling.
- Backbone Agnostic: The authors demonstrated that the "EARS boost" transfers across different VLM families, including Qwen2-VL, Qwen3-VL, and LLaVA-NeXT-Video.
- Efficiency: EARS adds negligible overhead. Pre-processing (motion/audio extraction) takes about 3 seconds per video on a single CPU, and the resulting evidence cache can be reused for any number of future queries.
Limitations and Future Work
- Stealthy Events: EARS fails on "stealthy" anomalies that don't produce motion or audio spikes (e.g., a person quietly shoplifting).
- Dense Narratives: In videos where the "evidence" is distributed throughout a long, complex story (like a movie or a tutorial), the motion-audio prior can be misleading, often focusing on irrelevant head-turns in egocentric footage.
- Future Direction: The authors suggest that the next logical step is adding a "semantic third channel" (like CLIP/SigLIP similarity) to the evidence score to handle cases where motion and sound are absent but visual meaning is high.
Summary updated