Large video-language models turn every sampled frame into hundreds of visual tokens. To keep inference affordable, a natural move is to discard tokens that duplicate one another. This proposal argues that the standard way of measuring duplication does damage. A moving hand looks almost the same in frame $t$ and frame $t+1$; it just sits a few pixels over. A redundancy pruner that compares token appearance will keep one copy and drop the other. In doing so it deletes the one piece of evidence that something moved. The pruned video then behaves like an unordered bag of distinct views, which amounts to a single image. That is the shortcut video benchmarks already reward too much.
We propose to show that appearance-based token pruning amplifies single-frame bias, meaning video models answering from the static content of one frame instead of from what changes over time. We then propose a fix. Redundancy should be judged against where a token was in the previous frame, not against what it looks like. A patch that is unchanged at the same location is waste. A patch that reappears at a displaced location carries motion and should be kept as a pair. Two further ideas size the token budget and refine the motion measurement. First, the variance of an average of positively correlated samples does not shrink with the number of samples, so the number of effectively independent frames in a clip sets how many tokens it deserves. Second, multi-scale band-pass filters, such as the Gabor banks used to model early primate vision, are sensitive to sub-patch displacements. Coarse transformer patches can blur those displacements.
Core Claim
The central claim is empirical and falsifiable. At matched token budgets, coverage-style redundancy pruning shrinks a model's ordered-versus-shuffled accuracy gap faster than it shrinks accuracy on static-answerable questions. A displacement-aware redundancy score preserves that gap, and with it accuracy on speed, direction and order questions.
The closest work, MMTok, casts visual token selection as submodular coverage: keep a set of tokens that jointly cover the query and the rest of the image. It retains 98.7% of LLaVA-NeXT-13B's POPE performance at a 1.87× speedup. It was built and evaluated on single images with a fixed budget, and it treats any two similar tokens as interchangeable. Our method differs in what counts as a duplicate, and in adapting the budget to each clip.
The distinction between harmful and useful overlap comes from Wu, Mao and Ng. They showed in static feature fusion that removing "relevant redundancy" (structured overlap that confirms a signal) hurts robustness, while removing "irrelevant redundancy" helps. We carry this distinction into time and give it a concrete test based on displacement.
Research Program
The first piece is a scoring rule. Take token embedding $x_{t,p}$ at frame $t$ and patch location $p$. Its static redundancy $s_{t,p}$ is its similarity to $x_{t-1,p}$, the token at the same location one frame earlier. Its motion evidence $m_{t,p}$ is high when the token is poorly matched at its own location but well matched at a nearby displaced location $p'$ in frame $t-1$.
The selection objective adds two terms to MMTok's coverage objective. One rewards keeping both ends of high-$m$ pairs. The other penalizes high-$s$ tokens beyond the first occurrence. Both terms are modular or coverage-like over the candidate set, so the greedy algorithm and its $(1-1/e)$ approximation guarantee carry over.
The second piece is a sharper motion measurement. We decompose each frame with a fixed multi-orientation, multi-scale quadrature Gabor bank of the kind VOneNet prepends to CNNs. That work showed such a front end confers robustness on static images. We compute the local phase shift between consecutive frames in each sub-band and pool it to patch resolution. This gives a second estimate of $m_{t,p}$ that can register displacements smaller than one patch.
The motivation is a limitation reported by Temporal Reasoning Transfer from Text to Video. That paper finds frozen encoders carry rich temporal signal (over 90% probe accuracy), yet its text-only fix fails on continuous motions such as rotation counts and fine speed differences. We do not assume Gabor phase is necessary. It is one of three motion estimators compared head to head; the other two are raw pixel differences and differences in the encoder's own features.
The third piece is the budget rule, drawn from the weighted-sum view. Let $w_t$ be the weight on frame $t$ and $\rho_{st}$ the correlation between pooled summaries of frames $s$ and $t$. The effective number of independent frames is
$$n_{\text{eff}} = \frac{\left(\sum_t w_t\right)^2}{\sum_{s,t} w_s w_t \rho_{st}}.$$
This follows from the variance expansion of a weighted sum with equal per-frame variances. When frames are near-identical, $n_{\text{eff}}$ approaches 1. When they decorrelate, $n_{\text{eff}}$ approaches $T$, the number of frames. We set the per-clip token budget proportional to $n_{\text{eff}}$, which addresses MMTok's fixed-budget limitation.
The role of the probability literature here is a lens, not a theorem we prove. The strong laws for weighted sums under mixing dependence in Hu, Wang and Wu formalize the principle that dependent samples average down only at the rate their dependence decays. Extra copies of a static background therefore buy almost nothing. We do not claim that visual embeddings satisfy their mixing conditions; the file notes that verifying such conditions for deep embeddings is open.
The method is training-free. It is applied at inference to an open Video-LLM: the Qwen2.5-VL-7B base used by Video-R1, plus one LLaVA-style video model such as TC-LLaVA.
Baselines
The do-nothing reference is the unpruned model run on all tokens from the same sampled frames. It sets the accuracy ceiling and the compute cost we are trying to cut.
Below it sits the single-frame reference. Here the model sees only the middle frame, mirroring the Image-LLM comparison in TempCompass, which found dedicated Video-LLMs often fail to beat a single-frame Image-LLM. Any pruning method whose ordered-minus-shuffled gap falls to this level has erased the video.
The naive pruning baseline is uniform frame subsampling at the matched token count. This is the cheapest way to meet a budget.
The strongest existing competitor is MMTok applied in two ways: per frame with the budget split evenly, and jointly over all frames' tokens. It is the strongest because it is training-free, has a guarantee, and is the best redundancy-aware pruner in the file. If coverage alone preserved temporal evidence, our method would be unnecessary. No reported result on Lacuna was found for MMTok on any video benchmark, so we run it ourselves using its released selection procedure.
A second redundancy baseline is greedy log-determinant selection from Selecting Diverse Features via Spectral Regularization. It picks maximally diverse tokens and is close in spirit to the repulsive sampling analyzed in the DPP Monte Carlo paper. It tests whether "more diverse" is the same as "more temporal."
All pruners share the same frames, encoder, language model and token budgets, set at 50%, 25% and 10% of the full count. Any score-mixing coefficient is tuned on one held-out split of a single benchmark with the same small grid for every method.
The ablations isolate the key ingredient. We drop the motion term while keeping the static penalty, drop the static penalty while keeping the motion term, swap the three motion estimators, and replace the $n_{\text{eff}}$ budget with a fixed budget. We beat the strongest baseline only if, at equal budget, we match MMTok on static-heavy benchmarks within one point and exceed it on the temporal subsets.
For reference points from the literature: TC-LLaVA reports 56.6% average accuracy on MVBench, and VideoChat2 with a Mistral-7B backbone reports 60.4% (MVBench). These are full-token numbers, and our pruned models are judged relative to their own unpruned version.
Why not just…?
Why not just fine-tune the language model on temporal text puzzles, as T3 does? T3 lifts Video-MME temporal reasoning from 37.3 to 49.7 without video training. This is complementary rather than competing. T3 cannot reason about motion tokens that pruning has already deleted, and it does nothing about token cost. We run T3-tuned and untuned backbones under each pruner. If pruning damage disappears once the backbone is T3-tuned, our claim is wrong and we will say so.
Why not just use MMTok-style coverage, since it already removes duplicates? This is the main threat, and it is directly tested. If joint-video MMTok preserves the ordered-versus-shuffled gap as well as our score does, the displacement test adds nothing. Our hypothesis predicts the opposite: coverage treats a moving object's consecutive tokens as duplicates. The shuffle-gap curve against pruning ratio settles this.
Why not just train against shuffling, as T-GRPO does? Video-R1 rewards the model when ordered frames beat shuffled ones. It reaches 37.1% on VSI-Bench against GPT-4o's 34.0%. It doubles training forward passes and was limited to 16 frames. It also says nothing about what to drop at inference. We apply our pruner to the released Video-R1 model. If our pruner lets that model use more frames at equal cost without losing its temporal gains, the two approaches compose.
Why not just measure motion with pixel differences or encoder-feature differences instead of Gabor filters? This may well work as well, which is why Gabor is one swappable estimator rather than the thesis. Pixel differences are cheaper but respond to lighting flicker and compression noise. Encoder-feature differences are free, but may be insensitive to sub-patch shifts. The controlled displacement experiment below decides between them. If encoder differences tie Gabor phase there, we drop the filter bank from the final method.
Proposed Experiments
The primary evaluation uses the speed, direction and event-order splits of TempCompass. There, current Video-LLMs already sit near random (25–33%) on speed and direction, so further collapse is easy to see. We add the 20 tasks of MVBench and the ATP-hard subset of NExT-QA from Revisiting the "Video" in Video-Language Understanding. No reported ATP-hard accuracy was found on Lacuna, so we report against our own unpruned model.
As a static-heavy control we use MSR-VTT multiple choice, where the single-frame ATP probe already reaches 93.2%. Here a good pruner should lose nothing.
For a retrieval sanity check we use SSv2-Template from SINGULARITY. There a single-frame model scores 42.0 R@1 against 77.0 for a temporal encoder. A pruned retrieval encoder should stay near the latter.
The first decisive check is the shuffle gap. For each method and budget we plot ordered-minus-shuffled accuracy on the temporal splits against accuracy on the static control. The hypothesis holds only if coverage and log-determinant pruning trace a curve that loses the gap first, while ours keeps it.
The second decisive check is a controlled synthetic test. Textured patches are rendered moving by 0.25 to 4 patch-widths per frame at known speeds and directions, and each motion estimator's pruned output is probed for speed and direction. This is where Gabor phase must beat encoder-feature differences at sub-patch displacements to earn its place.
The third check tests the budget rule. We correlate $n_{\text{eff}}$ with per-clip accuracy loss under fixed budgets. We then test whether $n_{\text{eff}}$-adaptive budgets reach the same average token count with smaller temporal losses. We report end-to-end latency, including selection overhead, against MMTok's reported sub-7 ms selection cost on images.
References
- MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
- Enhanced feature fusion through irrelevant redundancy elimination in intra-class and extra-class discriminative correlation analysis
- Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations
- Temporal Reasoning Transfer from Text to Video
- Strong convergence for weighted sums of (α, β)-mixing random variables and application to simple linear EV regression model
- Video-R1: Reinforcing Video Reasoning in MLLMs
- TC-LLaVA: Rethinking the Transfer of LLava from Image to Video Understanding with Temporal Considerations
- TempCompass: Do Video LLMs Really Understand Videos?
- Selecting Diverse Features via Spectral Regularization
- On two ways to use determinantal point processes for Monte Carlo integration
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
- Revisiting the "Video" in Video-Language Understanding
- Revealing Single Frame Bias for Video-and-Language Learning