A video model given only a few frames from a long video should spend that budget on frames that tell it something it could not have guessed. Current frame and token selectors measure redundancy in one of two ways. Some compare a frame to its immediate neighbour, by cosine similarity of features. Others reward frames that look different from one another. Both treat a smooth camera pan as a stream of new information, because every frame looks different, even though a simple motion model could predict each pan frame from the one before it. Both also miss the case where a frame looks similar to its neighbours but contains a small, unpredictable change that alters everything after it.
We propose to score frames with a controllability Gramian, a matrix from control theory that measures how strongly a set of inputs can push a dynamical system's state, and in which directions. We fit a small linear dynamics model to a video's frame features. Each frame's prediction error, the part the model could not foresee, is treated as an input injected into the system at that moment. We then choose the set of frames whose combined inputs, carried forward through the dynamics, span the largest volume of the state space. In short, frames are kept for being surprising in new directions, not for looking different.
Core Claim
Let $z_t \in \mathbb{R}^d$ be a compressed feature of frame $t$. Fit $z_{t+1} \approx A z_t$ by least squares on the video itself, and define the innovation $e_t = z_t - A z_{t-1}$, the part of frame $t$ the dynamics could not predict. For a candidate set $S$ of frames in a clip of length $T$, define the time-varying finite-horizon Gramian
$$W_S = \sum_{t \in S} A^{T-t} e_t e_t^\top (A^{T-t})^\top + \epsilon I.$$
Here each kept frame acts as an actuator placed at time $t$, and $\epsilon I$ is a small regularizer that keeps $W_S$ invertible. We select $S$ to maximize $\log\det W_S$ under a frame budget, with $\lambda_{\min}(W_S)$, the worst-covered direction, as a secondary criterion.
The claim is that this dynamics-aware, volume-based score beats both neighbour-similarity and plain diversity selection on long-video question answering at equal frame budgets. The gain should grow with video length and with the share of predictable motion in the video.
The formulation has a useful structure: its three ingredients can each be removed in turn.
- Removing the dynamics. Setting $A = 0$ makes $W_S$ a log-determinant of raw feature outer products. That is diversity sampling, the closest generic competitor.
- Removing the volume objective. Replacing $\log\det$ with the trace gives a per-frame score: the norm of each frame's innovation, propagated forward. By the modularity result of Optimal Sensor and Actuator Placement in Complex Dynamical Networks, which shows linear Gramian functions decouple across actuators, this is a sort-and-pick rule. It is close in spirit to codec-style predictive coding as in AdaCodec. The same paper notes that trace maximization piles actuators onto already-strong modes. In video, this predicts the trace rule will keep many frames of the same surprising event and drop a quieter, orthogonal one.
- Removing both. This reduces to uniform sampling.
Research Program
The method is training-free and sits in front of a frozen video large language model.
Feature extraction. Frames are encoded by the model's own vision encoder at a cheap candidate rate (for example 2 frames per second). Patch features are pooled and reduced to $d \approx 64$ by principal component analysis.
Dynamics fit. $A$ is fit per video by ordinary least squares. The choice is supported by Learning Without Mixing, which proves that least-squares error on correlated sequential data scales as $\sqrt{d/(T\,\lambda_{\min}(\Gamma))}$. With $d=64$ and hundreds to thousands of candidate frames per long video, the estimate is well posed. The same paper supplies a second motivation: excitation along the weakest Gramian direction is exactly what makes dynamics learnable. A selected set with large $\lambda_{\min}$ therefore gives a downstream model more information about how the scene evolves.
For long videos, we fit piecewise models on windows split at points where prediction error spikes. This borrows the decoupling idea of Controllability Metrics, Limitations and Algorithms for Complex Networks, which shows that a fixed number of actuators faces exponentially growing energy as a network grows, and that partitioning the system restores constant cost. Our hypothesis is that a single global model over an hour of video hits an analogous wall, and that segment-wise Gramians avoid it.
Selection. We select greedily, using rank-one determinant updates. Because $\log\det(\epsilon I + \sum_{t\in S} M_t)$ is monotone submodular for positive semidefinite $M_t$, greedy selection carries the standard $(1-1/e)$ guarantee. For $\lambda_{\min}$, we report the submodularity ratio and curvature empirically, following Performance guarantees for greedy maximization of non-submodular controllability metrics. That paper found greedy within 90% of optimal on random networks. We will check on short clips, by brute force, whether this holds for video-derived Gramians.
Adaptive budget. Greedy selection stops when the marginal log-determinant gain falls below a threshold. A static video then receives few frames and an eventful one many. This targets the fixed-budget limitation of Efficient Video Transformers with Spatial-Temporal Token Selection, whose top-K selector uses a fixed K.
Relation to prior selection work. Two parts of the control literature motivate how the method is structured.
- Heterogeneity of central nodes explains the benefits of time-varying control scheduling shows that moving actuators across time beats a fixed actuator set when different nodes dominate at different scales. Our selection is time-varying by construction, since each actuator is a moment in time. That paper's result is the reason to expect a Gramian over time to matter more than a static diversity score.
- Eigenvalue clustering, control energy, and logarithmic capacity proves that clustered system eigenvalues make $\lambda_{\min}$ exponentially small. We use it diagnostically: videos whose fitted $A$ has tightly clustered spectra should be the ones where selection matters least and redundancy is highest. We will test that correlation.
Composition with token pruning. Frame selection is one-shot and independent of the question. DyCoke argues directly against one-shot pruning. It shows that tokens judged unimportant early can become important later in decoding, and recovers them from a cache. We do not dispute this at the token level. Our method chooses which frames enter at all, and DyCoke can still run on the tokens of the kept frames. We test the combination explicitly. We also test a question-conditioned variant that adds a text-similarity weight to each $e_t$, in case question independence turns out to be the binding limitation.
Second regime: world-model keyframes. We test the method on robotic world-model keyframing. KeyWorld picks keyframes by running the Ramer–Douglas–Peucker algorithm on robot joint trajectories, generates only those frames, and interpolates the rest. This requires privileged kinematic data. Our selector uses only pixels, so the question is whether it can match the privileged choice.
Baselines
Uniform sampling. This is the naive reference: the video language model's default evenly spaced frames at the same budget. It is the right floor because ClipBERT showed sparse sampling is surprisingly strong, reaching MSRVTT Recall@1 of 22.0 against 16.8 for a dense-feature baseline. Any selector that fails to beat it at equal budget has no case.
DyCoke. This is the strongest existing training-free compressor for video language models. It merges tokens across adjacent frames by cosine similarity and dynamically restores pruned tokens during decoding. It reports an MVBench average of 57.5 at K=0.7 on LLaVA-OneVision, against 58.0 for full tokens, with a 1.54× speedup and 1.4× memory reduction on LLaVA-OV-7B. We run DyCoke's released settings on the same LLaVA-OneVision-7B checkpoint and match the number of visual tokens reaching the language model, not the number of frames. Beating it means higher accuracy at equal token count and equal wall-clock time, or equal accuracy at fewer tokens. On Video-MME, no reported result was found on Lacuna for DyCoke or for LLaVA-OneVision, so we will measure both ourselves.
Learned scorer. The spatial-temporal token selector of Efficient Video Transformers with Spatial-Temporal Token Selection uses a small trained network to rank frames. It reports 20–33% compute savings with under 0.7% accuracy loss on Kinetics-400. We include it on Kinetics-400 and Something-Something-v2 at matched compute, since a trained scorer is the obvious answer to any hand-designed criterion.
Diversity ablation ($A=0$). This is greedy log-determinant selection on raw features, a determinantal-point-process-style selector. It isolates whether the dynamics model adds anything. If it matches the full method, the control-theoretic framing is unnecessary.
Innovation-norm ablation (trace). This ranks frames by propagated innovation norm and isolates whether the volume objective matters beyond simple surprise. We tune both ablations with the same PCA dimension and budget grid as the full method.
Kinematic keyframes on LIBERO. The comparison point is KeyWorld's joint-trajectory keyframes, which reach 90% object-level accuracy on LIBERO-goal against 38% for frame-by-frame generation. We use the same diffusion transformer and interpolator and swap only the keyframe source. Matching within a few points without joint data counts as success. Beating it is not expected.
Why not just…?
Why not just pick frames that are maximally diverse in feature space? This is the most serious threat, and it is exactly our $A=0$ ablation. The argument against it is specific. Diversity selection rewards predictable change, such as pans, zooms, and steady walking, because consecutive frames differ in features even though the motion model predicts them. The Gramian score removes predictable change before measuring volume. If diversity selection ties with the full method on Video-MME long videos, and on a subset we tag for heavy camera motion, we will report that the dynamics term does not pay for itself.
Why not just keep the frames with the largest prediction error, like a video codec choosing reference frames? That is the trace ablation. Optimal Sensor and Actuator Placement in Complex Dynamical Networks identifies its failure mode: it concentrates selections along a few already strong directions. In video, that means many frames of one dramatic event at the expense of a quieter one. The comparison is decisive on multi-event questions, such as action sequence and ordering tasks in MVBench. There, $\log\det$ should win if the pathology is real.
Why not just use cosine similarity to the previous frame, as DyCoke does? Adjacent-frame similarity cannot tell slow, predictable drift apart from a small decisive change, and it has no view of the video as a whole. We do not frame this as a replacement. Our selector removes whole frames before tokenization and DyCoke compresses what remains, so the test is whether the combination beats DyCoke alone at matched token count.
Why not just train a scorer? A trained scorer can learn task-specific cues that our criterion cannot. It may well win on Kinetics-400, where scene appearance alone often identifies the action. Our method's advantages are that it needs no training, adapts its budget per video, and carries an approximation guarantee. We expect it to be competitive mainly on motion-centric Something-Something-v2. We will report honestly if the trained scorer dominates there too.
Proposed Experiments
Video question answering. The main experiment uses LLaVA-OneVision-7B on Video-MME at budgets of 8, 16, 32 and 64 frames, reporting short, medium and long splits separately. Video-MME documents up to a 14.3% accuracy drop from short to long videos even for Gemini 1.5 Pro. Our prediction is that the margin over uniform sampling and over diversity sampling grows from the short to the long split. We run the same protocol on MVBench, where clips are short. Gains there should be small and concentrated in temporal-order tasks.
Synthetic test of predictability. One check separates our hypothesis from plain diversity. We build synthetic and edited clips in which a long, smooth pan is spliced next to a brief, unpredictable event, such as an object appearing, and ask questions only about the event. Dynamics-aware selection should spend its budget on the event. Diversity selection should spend it on the pan.
Greedy optimality. On 30-frame clips we compute brute-force optima for small budgets and report how close greedy gets, alongside the empirical submodularity ratio.
Does the score predict accuracy? Across all videos and budgets, we test whether $\log\det W_S$ of the chosen set predicts question-answering accuracy better than the summed diversity score does. We also test whether eigenvalue clustering of the fitted $A$ predicts how little selection helps.
World-model keyframes. On LIBERO, we swap KeyWorld's joint-based keyframes for pixel-only Gramian keyframes, keeping everything else fixed.
Cost. We report the wall-clock overhead of the dynamics fit and greedy selection against the time saved in the language model. With $d=64$, the fit and rank-one updates should cost far less than encoding the frames, but we will measure this rather than assume it.
The proposal fails cleanly if the $A=0$ ablation matches the full method on long videos and on the synthetic test. That would show temporal redundancy in these models is about appearance, not predictability.
References
- Optimal Sensor and Actuator Placement in Complex Dynamical Networks
- AdaCodec: A Predictive Visual Code for Video MLLMs
- Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification
- Controllability Metrics, Limitations and Algorithms for Complex Networks
- Performance guarantees for greedy maximization of non-submodular controllability metrics
- Efficient Video Transformers with Spatial-Temporal Token Selection
- Heterogeneity of central nodes explains the benefits of time-varying control scheduling in complex dynamical networks
- Eigenvalue clustering, control energy, and logarithmic capacity
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
- KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Models
- Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark