Standard video models treat video as a uniform sequence of frames, processing every single image at a fixed rate (e.g., 30 frames per second). However, video data is highly temporally redundant. Consecutive frames often contain nearly identical visual information, and critical events are typically concentrated within brief, sparse moments. Forcing a heavy neural network to process every frame sequentially is computationally prohibitive, especially for long-form videos or edge devices. As Gou et al. and Wu et al. point out, feeding massive numbers of frames uniformly into a model's context window yields diminishing returns and incurs massive computational overhead.
Instead, humans understand video through information foraging—skimming through uninformative segments and focusing on critical transitions. This intuition drives the field of adaptive frame skipping (or selective temporal sampling). Rather than processing videos linearly, models learn a policy to dynamically choose which frames to inspect and how far to jump next, achieving massive computational savings with minimal loss in accuracy.
Framing Selection as a Sequential Decision Process
To build an agent that can "skim" a video, the problem is frequently modeled as a Markov Decision Process (MDP) solved via Reinforcement Learning (RL). At any given step, the agent observes the current frame (or a low-cost representation of it) along with its internal memory state, and then decides on an action: either emit a prediction or skip a certain number of frames.
[ Current Frame ] ---> ( Policy Network / RNN ) ---> Action: Skip N Frames
^ |
|__________________ Jumps Forward ___________________|
For instance, Yeung-Levy et al. pioneered this by training an agent to detect actions using "frame glimpses," processing less than 2% of the total frames. Similarly, Mahasseni et al. and Fan et al. formulated video navigation using discrete action spaces that allow the agent to jump forward by variable lengths or even rewind slightly when confidence drops. By moving non-linearly through the temporal dimension, these agents can pinpoint activity boundaries with extreme efficiency. In a similar vein, Mahasseni et al. utilize an LSTM-based policy to select keyframes for heavy-duty processing while using a lightweight network to interpolate the skipped frames, demonstrating that dense semantic segmentation does not require dense computation.
Dynamic Stride and Action Spaces
How exactly do these agents decide where to look next? The core mechanism relies on mapping the current visual context to a discrete or continuous action space representing temporal jump sizes.
In FFNet, Lan et al. utilize a deep Q-network to map features from the current frame directly to variable fast-forward step sizes, allowing the model to bypass irrelevant segments without decoding them. For dense video captioning, Suin et al. use an RL agent to dynamically adjust the temporal stride: the agent increases its sampling density during complex, high-activity events and switches to large skips during low-activity segments, reducing computational costs by 90%.
To capture multiple distinct events simultaneously, Wu et al. extend this to a multi-agent reinforcement learning framework where collaborative agents "walk" through the video timeline together, each tasked with identifying a specific discriminative segment. This ensures that the final selection captures the most relevant temporal information across the entire video.
Differentiable and Language-Guided Selection
While early methods relied heavily on policy gradients or Q-learning, modern architectures leverage differentiable modules or language guidance to align frame selection with specific semantic queries or complex reasoning tasks.
Meng et al. introduced a differentiable policy network in AR-Net that dynamically decides whether to process a frame at high resolution, low resolution, or skip it entirely, allowing the system to be optimized end-to-end. When the task involves reasoning or natural language queries, the selection mechanism can be conditioned directly on text. For example, Wang et al. use a semantic matching RL model to glimpse frames based on a text query, achieving a six-fold speedup in activity localization.
In the era of Large Vision-Language Models (VLMs), Buch developed the Flexible Frame Selector (FFS), a lightweight learnable module that identifies exactly which frames a VLM needs to answer a specific question, slashing compute by up to 13.5x. Similarly, Yan et al. utilize a text-guided frame sampler to capture long-term video context for complex reasoning video object segmentation, while Xu et al. demonstrate that an adaptive frame selection mechanism can navigate forward and backward to capture critical causal events in complex traffic environments.
Why It Works: Exploiting Information Triggers
The fundamental reason adaptive frame skipping works so well without sacrificing accuracy is that video information density is highly non-uniform. A model does not need to see every frame of a drone flight, a traffic intersection, or a movie scene to understand what happened; it only needs to see the state transitions.
Uniform Sampling: [X][X][X][X][X][X][X][X][X][X][X][X] (High Cost)
Adaptive Skipping: [X]--------->[X]----->[X]--------->[X] (Low Cost)
^ ^ ^ ^
Start Turn Left Collision End
Farhadi demonstrates this in Temporal Knowledge Distillation (TKD), where a recurrent selection module identifies high-information frames—such as scene transitions—to trigger online model updates, ignoring redundant frames that offer no new environmental information. In aerial navigation, Gao et al. show that action-triggered keyframe selection successfully filters out redundant visual data by focusing exclusively on critical maneuvers and landmark appearances. For complex spatiotemporal reasoning involving object permanence, Zhou et al. show that a Multi-hop Transformer can successfully track hidden objects by iteratively "hopping" through critical transition points rather than processing uniform video segments.
By treating temporal sampling as a dynamic, content-dependent choice rather than a static preprocessing step, these architectures achieve the accuracy of heavy-duty models at a fraction of the computational cost.
Key Foundational Work
End-to-End Learning of Action Detection from Frame Glimpses in Videos (2016) — contributed 1 concept elements to this direction.
Budget-Aware Deep Semantic Video Segmentation (2017) — contributed 1 concept elements to this direction.