A standard video recorded at 30 frames per second contains an immense amount of data, but very little actual information changes from one millisecond to the next. If an actor walks across a room, the background remains identical across hundreds of frames, and the actor's posture shifts only incrementally.
When traditional deep learning architectures treat every pixel or video frame as an independent, equally important piece of information, they waste massive amounts of computational power. Processing static backgrounds or repetitive motions over and over again inflates the key-value (KV) cache in Transformers, causes quadratic complexity explosions in attention mechanisms, and introduces semantic noise that can distract a model from the core action.
To build efficient video models, researchers focus on exploiting temporal redundancy. By identifying and eliminating repetitive data across time, models can achieve massive speedups and memory reductions while maintaining—or even improving—accuracy. This paradigm operates across three main levels: frame-level sampling, token-level compression, and direct compressed-domain processing.
Frame-Level Sparsity: Sampling and Key-Framing
The most intuitive way to handle temporal redundancy is to avoid looking at every frame in the first place. Dense sampling creates significant computational overhead, yet much of the temporal information is completely unnecessary for understanding a scene or establishing semantic links with text.
Sparse Sampling
Instead of processing an entire video end-to-end, the authors of ClipBERT demonstrated that training models on just one or a few short, sparsely sampled clips is sufficient for video-and-language learning. This "less is more" approach drastically reduces the computational footprint, allowing the visual backbone to be fine-tuned directly on the target task rather than relying on fixed, offline-extracted features. Similarly, Song showed that a tiny training set (as small as 2% of the total data) carefully selected for temporal and spatial diversity can significantly outperform larger, randomly sampled datasets by eliminating redundant labeling and processing efforts.
Semantic Key-Framing
Not all frames are created equal. In robotic world models, generating frames at a fixed temporal frequency forces the model to expend high-level reasoning resources on low-level visual continuity. To solve this, Hao developed KeyWorld, a framework that shifts the focus from frame-by-frame generation to reasoning exclusively about semantically critical "key frames" (such as the exact moment a robot grasps an object). A lightweight interpolator then fills in the redundant gaps, accelerating the world model by over 5×.
Dense Frame Stream: [F1] -> [F2] -> [F3] -> [F4] -> [F5] -> [F6] (High Redundancy)
Sparse Key-Framing: [F1] -------------------------> [F6] (Semantic Transition)
\__ Linear Interpolation __/
This concept of isolating defining moments also underpins action recognition. Raptis et al. proposed representing human activities as a sparse sequence of latent keyframes composed of mid-level poselets, making the model robust to varying action durations and irrelevant, slow-moving frames.
Token-Level Compression: Pruning and Merging
When a model must process a sequence of frames, treating every spatial patch across time as a unique token quickly overwhelms attention mechanisms. In modern Video Large Language Models (VLLMs) and Vision Transformers (ViTs), researchers exploit temporal redundancy by dynamically compressing these tokens on the fly.
Token Merging
Because adjacent frames share nearly identical visual information, their corresponding visual tokens are highly similar. Tao et al. introduced DyCoke, a training-free method that merges highly similar visual tokens across consecutive video frames during the prefilling stage. By consolidating static or slow-changing visual features into a single representation, it reduces the initial token count by over 50% and prevents duplicate tokens from unnecessarily inflating the KV cache.
Dynamic Token Selection and Pruning
Instead of keeping all tokens, models can actively drop redundant ones. Wang et al. developed STTS (Spatial-Temporal Token Selection), demonstrating that temporal redundancy in video datasets is significantly higher than spatial redundancy. This insight allows for aggressive frame-level pruning with minimal impact on classification accuracy.
Other architectures apply this concept via graph structures or attention masks:
- SViTT: Min et al. use graph-inspired sparsity to prune redundant visual patches (nodes) and limit attention connections (edges), cutting down memory and FLOPs.
- SwinBERT: Lin et al. utilize a learnable Sparse Attention Mask to filter out static backgrounds and repetitive motions, preventing the attention mechanism from being overwhelmed by noise.
- EcoCued: Liu et al. introduce a Token-Importance-Aware Attention mechanism that slashes computational complexity from quadratic $O(T^2)$ to linear $O(T)$ by identifying and down-weighting linguistically redundant frames.
Bypassing Pixels: The Compressed Domain and Cache Sharing
Beyond sampling and pruning, some frameworks eliminate temporal redundancy by changing how the video data is represented or cached during computation.
Processing Compressed Streams
Traditional video models waste significant computation decoding compressed video files (like MP4s) back into raw, redundant RGB frames. Wu et al. introduced CoViAR, a framework that trains deep networks directly on the compressed video stream itself.
Raw Video Decoding: [Compressed File] -> [Decompress to RGB Frames] -> [Heavy 3D CNN]
|
Massive Spatial Redundancy
Compressed Processing: [Compressed File] -> [Motion Vectors & Residuals] -> [Lightweight CNN]
|
Naturally Filtered Signal
Because video compression algorithms already store data as motion vectors and residuals (only saving the changes between frames), the compressed stream functions as a pre-filtered signal. It strips away the vast majority of redundant spatial information, allowing the neural network to focus entirely on the core temporal features that define an action.
Cache Sharing in Generative Models
In autoregressive video generation, such as video diffusion models, standard architectures often utilize bidirectional attention. This requires the model to re-compute features for overlapping context frames over and over again across different segments.
To eliminate this redundant computation, Gao et al. developed Ca2-VDM. By implementing causal generation and cache sharing, the model retains features from previous steps and shares them across segments. This simple shift turns a quadratic computational complexity growth into a linear one, drastically reducing memory overhead during long-form video generation.
Why It Works: Information Density
Exploiting temporal redundancy works because it increases the information density of the data flowing through a network. Raw video is a low-density signal dominated by static pixels and repetitive frames. By forcing a model to process every pixel uniformly, we compel it to spend its representational capacity learning to ignore irrelevant background details.
Whether by sampling keyframes, merging adjacent tokens, or reading motion vectors directly from compressed streams, removing temporal redundancy allows neural networks to focus their parameters and FLOPs purely on the dynamic changes that contain the actual semantic meaning of the video.