Towards Sparse Video Understanding and Reasoning

Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

In the world of computer vision, video is the ultimate "big data" challenge. It’s high-dimensional, temporally redundant, and semantically dense. Most current Vision-Language Models (VLMs) handle video by uniformly sampling frames—say, one frame per second—and shoving them all into a massive prompt.

The result? Informational Overload. Models get overwhelmed by redundant frames and often miss the "needle in the haystack" frames critical for answering a specific question. A new paper from researchers at Northwestern, Johns Hopkins, and Dolby Laboratories proposes a more elegant solution: REVISE (Reasoning with Video Sparsity).

TL;DR

REVISE is a multi-round agent that treats video understanding as an iterative search rather than a single-shot processing task. By maintaining a concise "summary-as-state" and selectively requesting only the most informative frames, it achieves state-of-the-art accuracy on video QA benchmarks while using significantly fewer frames and tokens.

What Problem Does This Paper Address?

Current VLMs suffer from two primary issues in video tasks:

  1. Informational Overload (L1): Long videos are full of "filler." Processing every frame wastes computation and can actually confuse the model’s reasoning.
  2. Insufficient Key Information Awareness (L2): Because models often process a fixed, uniform set of frames, they might miss a split-second action (like a key being turned or a specific gesture) that is essential to answering a question.

The authors argue that video understanding should be sparse. Only a tiny fraction of frames are usually relevant to a specific query. The challenge is finding them without looking at everything first.

Methods

REVISE functions as a multi-round agent. Instead of seeing the whole video at once, it follows a loop:

  1. Iterative Selection: The agent starts with a few frames, then "asks" for specific timestamps it thinks will help reduce uncertainty.
  2. Summary-as-State: Inspired by the hidden states in Recurrent Neural Networks (RNNs), REVISE maintains a structured textual summary that is updated every round. This summary tracks:
  3. P (Previously seen frames)
    • O (Observations)
    • H (Belief updates/Hypotheses)
    • U (Remaining uncertainties)
    • R (Reasons for the next frame request)
  4. Early Stopping: Once the agent is confident it has the answer, it stops, saving on API costs and compute.

For open-source models, the authors introduced EAGER (Evidence-Adjusted Gain for Efficient Reasoning), a reinforcement learning (RL) reward system. EAGER rewards the model for increasing its confidence in the correct answer, maintaining a "sufficient" summary that can answer the question even without the original frames, and stopping early.

Datasets

The researchers evaluated REVISE across several demanding benchmarks:

  • VideoEspresso: A large-scale dataset focused on fine-grained, chain-of-thought video reasoning.
  • NExT-QA: Targets causal and temporal action reasoning (e.g., "Why did X happen?").
  • EgoSchema: A long-form egocentric video benchmark with 3-minute clips.
  • Video-MME & LVBench: Recent benchmarks for long-video understanding.

Key Insights and Ideas

  • Text as Memory: One of the most clever aspects is using a structured textual summary as the "state." By forcing the model to write down its uncertainties and reasons, the researchers found the reasoning process became much more stable and interpretable.
  • The Pareto Frontier of Reasoning: The paper demonstrates a clear trade-off: as you allow the model more "turns" to request frames, accuracy goes up while the total number of frames used actually goes down. The model becomes a "sniper" rather than using a "shotgun" approach.
  • Annotation-Free RL: The EAGER reward doesn't require humans to label which frames are important. It simply looks at whether the model's internal probability for the correct answer increases after seeing new frames.

Key Results

The performance gains are substantial, especially for proprietary models like GPT-4o:

  • Massive Accuracy Boost: On the VideoEspresso benchmark, GPT-4o + REVISE improved average accuracy from 26.4% to 48.9%—a staggering +22.5% jump—while using only about 8 frames per video.
  • Efficiency: For open-source models like Qwen2-VL, REVISE improved accuracy by nearly 10% while keeping the frame count in the single digits.
  • RL Success: Using RL fine-tuning (REVISE-RFT), a 3B-parameter model (Qwen2.5-VL) was able to outperform much larger models by learning to be more "decisive" and selective with its evidence gathering.

Limitations and Future Work

While REVISE is a major step forward, the authors acknowledge a few hurdles:

  • Backbone Dependence: The agent is only as good as the VLM it "wraps" around. If the underlying model has poor visual fidelity, the summary state will eventually degrade.
  • Interaction Latency: Because REVISE is multi-round, it requires multiple sequential API calls or inference passes. In a real-time system, this latency might be a bottleneck compared to a single-shot model.
  • Future Directions: The authors suggest exploring more complex "actions" for the agent, such as zooming into specific spatial regions of a frame or using external tools to verify evidence.

REVISE proves that in the era of massive context windows, the smartest way to handle data isn't always to "see it all," but to know exactly what to look for.

Summary updated

Abstract

We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA). Instead of uniformly sampling frames, \revise selects a small set of informative frames, maintains a summary-as-state across rounds, and stops early when confident. It supports proprietary vision-language models (VLMs) in a ``plug-and-play'' setting and enables reinforcement fine-tuning for open-source models. For fine-tuning, we introduce EAGER (Evidence-Adjusted Gain for Efficient Reasoning), an annotation-free reward with three terms: (1) Confidence gain: after new frames are added, we reward the increase in the log-odds gap between the correct option and the strongest alternative; (2) Summary sufficiency: at answer time we re-ask using only the last committed summary and reward success; (3) Correct-and-early stop: answering correctly within a small turn budget is rewarded. Across multiple VQA benchmarks, \revise improves accuracy while reducing frames, rounds, and prompt tokens, demonstrating practical sparse video reasoning.
BibTeX
@misc{xu2026towards,
  author = {Chenwei Xu and Zhen Ye and Shang Wu and Weijian Li and Zihan Wang and Zhuofan Xia and Lie Lu and Pranav Maneriker and Fan Du and Manling Li and Han Liu},
  title = {{Towards Sparse Video Understanding and Reasoning}},
  year = {2026},
  eprint = {2602.13602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.13602},
}
Figure