Coarse-to-Fine Temporal Localization in Long Videos

80 concepts65 papers1 proposal

The "needle in a haystack" problem defines the current challenge of video understanding: as video lengths grow from seconds to hours, the computational cost of processing every frame at high resolution becomes prohibitive. To address this, researchers have converged on a coarse-to-fine paradigm. By mimicking human visual search—skimming a sequence to find relevant areas before "zooming in" for details—these methods decouple global context modeling from local precision, enabling efficient localization in untrimmed, long-form content.

The Efficiency Bottleneck

Traditional video models often treat sequences as monolithic inputs, applying dense feature extraction across the entire temporal axis. However, in real-world applications like movie retrieval or surveillance, the vast majority of frames are redundant or irrelevant. Laptev et al. recognized this early on, arguing that heuristic search space reduction—such as using a fast detector to prune unlikely regions—is essential to avoid the exhaustive evaluation of every possible spatio-temporal window.

This bottleneck is particularly acute in long-form video. Park et al. demonstrate that a hierarchical selection strategy can filter out over 98% of redundant frames before they ever reach an expensive vision-language model. By discarding the "haystack" early, systems can maintain high accuracy while reducing costs by an order of magnitude.

The Skim-and-Zoom Mechanism

The core of the coarse-to-fine approach is a two-stage pipeline that separates candidate generation from boundary refinement.

  1. Stage 1: The Global Skim. A lightweight model scans the video at low resolution or a high frame rate to identify broad "candidate windows." For example, Wu (LiteEval) uses a lightweight network to skim all frames, selectively invoking a heavy-duty model only for high-resolution analysis of informative segments. Similarly, Liang et al. (S2DNet) decouple coarse scene scanning from fine-grained detail extraction, using a "synopsis" to determine where and when a high-capacity network should focus its resources.
  2. Stage 2: The Local Zoom. Once a candidate window is identified, the system applies intensive processing to pinpoint exact start and end times. Qin et al. (TimeScope) use this to navigate videos over an hour long, first identifying a broad window and then using high-resolution data to ground specific tasks. Zhong et al. (CONE) show that this hierarchical alignment can speed up inference by 15x compared to standard sliding-window methods.

Hierarchical Pruning and Triage

To scale to massive datasets, the coarse stage often acts as a "triage" system. Iftekhar et al. (DeCafNet) formalize this as a "delegate-and-conquer" strategy, where a lightweight "sidekick" encoder triages video clips, passing only the most relevant ones to a heavy "expert" encoder.

This tiered approach is also effective for multi-modal tasks. Shao et al. (FIFO) manage large video databases by first filtering candidates at a global level using visual-semantic embeddings and then applying fine-grained temporal localization only to the top results. Even in specialized domains like news broadcasts, Shearer et al. found that identifying coarse structural markers (like anchor shots) allows a system to selectively apply expensive techniques like face recognition only where they are likely to succeed.

Recent work has evolved the two-stage pipeline into more iterative, agentic processes. Wu et al. (VideoAgent2) introduce an uncertainty-aware Chain-of-Thought process where an LLM agent iteratively plans and retrieves information, adjusting its search strategy based on confidence scores.

Some researchers have even reframed temporal search as a spatial problem. Ye et al. (T*) arrange video frames into image grids and iteratively "zoom in" on promising regions, achieving a logarithmic reduction in computational complexity. This progressive refinement allows models to handle the vast context of long videos without exceeding memory limits, as noted by Li et al. (UniTime), who use a coarse-to-fine strategy to localize events in videos spanning several hours.

Precision through Decoupling

Beyond efficiency, the coarse-to-fine strategy improves localization precision by decoupling distinct sub-tasks. Ju and Zhang argue that isolating action instances into individual clips before localizing them prevents the performance trade-offs inherent in one-stage methods. Similarly, Yang separates the search for entities (the "who") from motion (the "how"), using the first stage to filter for context and the second to pinpoint temporal boundaries based on dynamics.

By breaking the localization task into manageable, hierarchical steps, these frameworks allow AI to navigate the complexity of real-world video with the same selective focus as the human eye.

Key Foundational Work

Retrieving actions in movies (2007) — contributed 2 concept elements to this direction.
Incorporating Domain Knowledge with Video and Voice Data Analysis in News Broadcasts (2000) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_coarse_to_fine_temporal_localization_d9b17e5d,
  title = {coarse-to-fine temporal localization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/coarse-to-fine-temporal-localization/rdr_d9b17e5dd4924971b823a328f98925a9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_coarse_to_fine_temporal_localization_d9b17e5d_v1,
  title = {coarse-to-fine temporal localization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/coarse-to-fine-temporal-localization/rdr_d9b17e5dd4924971b823a328f98925a9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake