Coarse to Fine Reasoning for Long Video Understanding

This direction is split into other directions: coarse-to-fine temporal localization. Its last version stays readable and citable here.

Version 1 of 1195 papers237 conceptsfrozen 2026-10-04

Understanding a video that lasts minutes or hours is fundamentally different from understanding a five-second clip. In short clips, a model can afford to process every frame at high resolution. In long-form video, this approach hits a "Sampling Dilemma": sparse sampling misses the "needle in the haystack" (fine-grained details), while dense sampling creates a "memory explosion" that exceeds the context limits of even the largest Multimodal Large Language Models (MLLMs).

As identified by Tianyuan Qu and colleagues in Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?, the solution is not more memory, but better allocation. A new wave of research proposes a coarse-to-fine paradigm: mimicking human visual search by first "skimming" the entire video to find relevant segments and then "zooming in" to perform dense reasoning only where it matters.

The Skim then Zoom Mechanism

The core intuition behind this research direction is that most frames in a long video are redundant for a specific query. Instead of treating the video as a flat sequence of tokens, models like Zoom-Zero, developed by Xiaoqian Shen and others, treat video understanding as a two-stage navigation process.

  1. The Glance (Coarse): The model scans the entire video using sparse global sampling or low-resolution previews. This stage builds a "map" of the video’s content.
  2. The Focus (Fine): Once a relevant temporal window is identified, the model re-allocates its entire computational "token budget" to that specific segment, processing it at high resolution to extract the "smoking gun" evidence.

This "Glance and Focus" strategy, as framed by Ziyi Bai in Glance and Focus: Memory Prompting for Multi-Event Video Question Answering, allows models to maintain global context without sacrificing the fine-grained detail necessary for complex reasoning.

Hierarchical Video Representations

To make this search efficient, researchers are moving away from linear lists of frames toward hierarchical structures. Ziyang Wang and the team behind VideoTree represent long videos as adaptive trees. This allows an LLM to "walk" down branches—moving from a 10-minute chunk to a 30-second segment—only expanding the nodes that are semantically relevant to the user's question.

Similarly, Ziqi Pang and Yu-Xiong Wang propose MR. Video, which applies the MapReduce principle to video. By "mapping" short clips to independent perception modules and "reducing" them into a global summary, they bypass context limits entirely. This hierarchical filtering can remove over 98% of redundant frames while actually increasing accuracy, as demonstrated by the LVNet framework from Jongwoo Park.

Agentic Navigation and Tool Use

The most recent shift in this field involves treating the MLLM as an active agent rather than a passive processor. In the VITAL framework by Chixiang Ma and colleagues, the model "thinks with videos" by dynamically calling visual tools to sample frames on demand.

This agentic approach introduces several key mechanisms:

  • Uncertainty-Aware Planning: Qiangqiang Wu and the VideoAgent2 team use confidence scores to let the agent decide when it needs to "backtrack" or look closer at a segment it previously skimmed.
  • Iterative Reasoning: Models like LOVE-R1 by Shenghao Fu treat video understanding as a multi-step conversation where the model asks itself, "Based on this low-res scan, where should I zoom in next?"
  • Reinforced Grounding: By using Reinforcement Learning (RL), as seen in Video-in-the-Loop by Chendong Wang, models learn to optimize their "glance" to maximize the success of their "focus."

Why It Works: Efficiency through Sparsity

The success of these methods—often outperforming massive models like GPT-4o on long-video benchmarks—stems from a fundamental realization: temporal grounding is a search problem, not just a perception problem.

By using a lightweight selector network or a "Generative Frame Sampler" (Linli Yao et al.) to narrow the search space, these systems ensure that expensive multimodal reasoning is only applied to high-information regions. As Xiangteng He shows with SOONet, scanning a video once to generate coarse anchors and then refining them is up to 100x more efficient than traditional sliding-window approaches. This shift from "process everything" to "search and refine" is what finally enables AI to reason over hours of footage on consumer-grade hardware.

Papers in this version

Showing 60 of 195 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_coarse_to_fine_reasoning_for_long_video__8d2f2b42,
  title = {Coarse to Fine Reasoning for Long Video Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/coarse-to-fine-reasoning-for-long-video-understanding/rdr_8d2f2b42fa8e497ea7f26949aceb66ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_coarse_to_fine_reasoning_for_long_video__8d2f2b42_v1,
  title = {Coarse to Fine Reasoning for Long Video Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/coarse-to-fine-reasoning-for-long-video-understanding/rdr_8d2f2b42fa8e497ea7f26949aceb66ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}