Eliminating Dense Proposals for Efficient Video Grounding

77 concepts76 papers1 proposal

When we watch a video to find a specific moment—such as "the exact second a magician flips a card"—our brains do not perform a brute-force, frame-by-frame comparison. We skip ahead, look for contextual cues, and quickly narrow down the window.

Yet, traditional computer vision models have long approached video grounding and temporal action localization through an exhaustive, computationally heavy paradigm known as propose-and-rank. To find an event, these models generate thousands of overlapping candidate segments across various scales using sliding windows, extract features for each, and rank them against a query. As video datasets scale from short clips to hours of unscripted footage, this exhaustive search strategy creates a massive computational bottleneck.

To overcome this scalability crisis, researchers have shifted toward proposal-free architectures. By rethinking temporal search as direct boundary regression, parallelized multi-dimensional scoring, or dynamic navigation, modern frameworks eliminate proposal redundancy and achieve dramatic speedups without sacrificing accuracy.

The Propose-and-Rank Bottleneck

To understand why traditional methods fail to scale, consider the mathematics of a video timeline. If a video contains $N$ frames, the number of possible temporal intervals grows quadratically, at $O(N^2)$.

As Jiawei Tan et al. point out in Anchor-Aware Similarity Cohesion in Target Frames Enables Predicting Temporal Moment Boundaries in 2D, evaluating every potential interval via brute force is highly inefficient. In two-stage architectures, an initial network generates dense proposals, and a second network ranks them. Runhao Zeng et al. observe in Dense Regression Network for Video Grounding that this design introduces severe limitations: inference speed is bottlenecked by the sheer volume of candidates, and the model's final accuracy is strictly capped by the quality and coverage of the initial proposals.

Traditional Propose-and-Rank:
[Untrimmed Video] ──> [Generate Thousands of Overlapping Proposals] ──> [Cross-Modal Ranking] ──> [NMS Post-Processing]
                                                                             │
                                                                     (High Latency Bottleneck)

This redundancy isn't limited to the temporal dimension. In tasks like video instance segmentation, processing hundreds of region proposals per frame results in substantial computational waste because the vast majority of candidates are eventually discarded, as noted by Dongfang Liu in SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation.

Direct Boundary Regression: Going Proposal-Free

The core intuition behind proposal-free grounding is to treat the video timeline similarly to how modern Natural Language Processing handles extractive question answering. Instead of scoring pre-defined text chunks, a model predicts the exact start and end indices directly from a global representation.

Soham Ghosh et al. pioneered this shift with ExCL: Extractive Clip Localization Using Natural Language Descriptions, demonstrating that models can directly regress boundary points rather than relying on fixed-length candidate windows. By mapping the natural language query into the video's temporal space, the model outputs probability distributions for the start and end timestamps across the entire video sequence in a single pass.

Proposal-Free Direct Regression:
[Untrimmed Video + Text Query] ──> [Global Cross-Modal Attention] ──> [Direct Start/End Timestamp Regression]

To refine this process, Cristian Rodríguez-Opazo introduced guided attention mechanisms and "soft" labels in Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention to better account for human timing uncertainty during boundary annotation.

With the advent of Transformers, this direct mapping became even cleaner. For instance, Fengyuan Shi et al. developed a parallel regression framework in End-to-End Dense Video Grounding via Parallel Regression that treats language embeddings as queries that directly decode boundaries, completely bypassing hand-crafted post-processing steps like Non-Maximum Suppression (NMS). Similarly, Kun Li et al. utilized a dedicated, learnable regression token within the Transformer architecture in ViGT: proposal-free video grounding with a learnable token in the transformer to aggregate global context and output precise temporal coordinates efficiently.

Parallel Dense Scoring: 2D Temporal Maps

If a model must evaluate multiple potential intervals to capture complex actions, doing so iteratively is too slow. A second major strategy solves this by evaluating all possible temporal intervals simultaneously using multi-dimensional maps.

In BMN: Boundary-Matching Network for Temporal Action Proposal Generation, Tianwei Lin et al. structured densely distributed proposals into a unified 2D confidence map. One axis represents the starting time of an action, while the other represents its duration. Through a specialized Boundary-Matching layer, the network evaluates the confidence scores of all potential action proposals at once, converting an iterative search problem into a highly parallelized, single-stage tensor operation.

2D Boundary-Matching Map:
  Duration
     ▲
     │  [X] (High Confidence Action)
     │     [ ]
     │        [ ]
     └────────────────► Start Time
(All pairs scored simultaneously via parallel tensor operations)

Daizong Liu et al. expanded on this concept in Context-Aware Biaffine Localizing Network for Temporal Sentence Grounding by employing a biaffine mechanism to score all possible start-end frame pairs simultaneously. This allows the network to seamlessly integrate multi-scale local and global contexts, making it easier to distinguish between visually similar actions across the timeline without incurring the latency of sequential proposal generation.

Dynamic Navigation: Iterative Search and Reinforcement Learning

A third paradigm entirely abandons linear frame processing. Instead of looking at the whole video at once or generating static windows, these methods deploy intelligent agents that dynamically navigate the video timeline.

In Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos., Dongliang He et al. framed temporal grounding as a reinforcement learning (RL) problem. The agent "reads" the text query, "watches" small, downsampled glimpses of the video, and dynamically decides whether to shift its temporal window left, right, or expand/shrink it.

Dynamic RL Navigation:
[Query] ──► [Agent Glimpses Frame t] ──► Action: [Shift Window Right] ──► [Glimpses Frame t+20] ──► Action: [Emit Boundaries]

This iterative refinement process mimics human intuition. Rather than processing every single frame, the agent skips irrelevant background content entirely. Jingjia Huang et al. demonstrated in SAP: Self-Adaptive Proposal Model for Temporal Action Detection Based on Reinforcement Learning that such self-adaptive agents can locate actions at hundreds of frames per second precisely because they prune the search space on the fly based on video content.

This concept of non-linear search reaches its logical extreme in long-form video understanding, where videos span hours. Jinhui Ye et al. introduced T in Re-thinking Temporal Search for Long-Form Video Understanding, a framework that completely breaks the linear $O(N)$ temporal processing paradigm. Instead of scanning a long video sequentially, T** arranges video frames into hierarchical 2D image grids and treats temporal localization as an iterative spatial "zoom-in" task. By converting temporal search into a multi-scale spatial localization problem, the framework isolates relevant segments in long-form videos with a fraction of the computational overhead of traditional models.

Why It Works: A Paradigm Shift in Efficiency

The transition away from exhaustive sliding windows and dense proposals works because it aligns a model's computational budget with the information density of the task.

  1. Mathematical Optimization: Moving from $O(N^2)$ interval evaluation to direct boundary regression or parallelized 2D mapping eliminates redundant feature extraction for highly overlapping segments.
  2. End-to-End Trainability: Proposal-free networks optimize a single loss function directly tied to the final timestamps, removing the misalignment between a proposal generator's recall and a ranking network's precision.
  3. Sparse Processing: Dynamic navigation and grid-based zooming allow models to ignore hours of irrelevant background frames, focusing heavy cross-modal attention mechanisms exclusively on the most promising visual regions.

By treating video timelines as continuous spaces to be regressed or dynamically navigated—rather than discrete segments to be exhaustively counted—proposal-free architectures provide the scalability required for real-time, long-form video intelligence.

Key Foundational Work

Modeling the Temporal Extent of Actions (2010) — contributed 1 concept elements to this direction.
Finding people in archive films through tracking (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sliding_window_computational_overhead_8775d94e,
  title = {sliding window computational overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sliding-window-computational-overhead/rdr_8775d94ea1e64757b6ea96e7a1a42ff2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sliding_window_computational_overhead_8775d94e_v1,
  title = {sliding window computational overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sliding-window-computational-overhead/rdr_8775d94ea1e64757b6ea96e7a1a42ff2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake