Power-Law Memory Tokens for Video Language Models Under a Fixed Cache Budget

Video language models read a video as a long sequence of image patches, called visual tokens, and keep each one in a key-value cache. This cache is the model's working memory: every new token or answer word attends back over it. Because consecutive frames look nearly identical, most of these tokens are redundant. Current compression methods therefore merge or evict them. Once a token is evicted it is gone, or at best it sits in a side buffer until someone decides to look at it again. We propose a different treatment. Evicted tokens are folded into a small, fixed set of summary tokens. Each summary token is a weighted average of everything evicted so far, and the weights decay with age as a power law (roughly $1/\text{age}^{\alpha}$) rather than exponentially. The model keeps full-resolution tokens for the recent past and a fixed number of summary tokens that still carry faint but non-zero traces of the distant past. The frozen model can attend to them like any other token.

Why a power law? Physical and control systems with long-range memory, such as porous-electrode batteries, elastic water columns, and epidemics, are described much better by fractional-order dynamics than by exponential ("integer-order") models. Fractional dynamics are differential equations whose memory kernel is a power law rather than an exponential. Two recent machine-learning results suggest the same holds for learned sequence memory. FRACTAL argues that exponential memory cannot retain the full history while also staying sensitive to recent input and invariant to time scale. Fractional-Order Spiking Neural Network shows that power-law neurons bridge dropped frames better than exponentially leaking ones. Streaming video question answering, where a question about minute two arrives at minute twenty, is exactly a problem of retaining the full history while staying recency-sensitive. The claim is testable at a fixed token budget, and it could fail.

Core Claim

Consider a frozen video language model running under a hard cache budget. Replace discarded tokens with $K$ summary tokens per memory slot, each built with a fractional power-law kernel of a different order. We claim this yields higher question-answering accuracy on questions about temporally distant events than three alternatives at the same token count: dropping the tokens, buffering them for recall, or summarizing them with exponential or uniform averages. Accuracy on recent events should be unchanged.

The distinctive, falsifiable signature concerns accuracy as a function of how far back the answer-relevant event lies. With power-law memory, accuracy should decay slowly and smoothly with that distance. With exponential memory, accuracy should fall off sharply past the chosen time constant. With a uniform average, recent detail should be blurred instead.

Concretely, for memory slot $i$ and kernel order $\alpha_k \in (0,1)$, the summary token at time $t$ is

$$m_{i,k}(t) = \frac{\sum_{\tau \in E_i(t)} (t-\tau+\Delta_0)^{-\alpha_k}\, x(\tau)}{\sum_{\tau \in E_i(t)} (t-\tau+\Delta_0)^{-\alpha_k}}.$$

Here $E_i(t)$ is the set of frame times whose evicted tokens were assigned to slot $i$, $x(\tau)$ is an evicted token's hidden vector, and $\Delta_0$ is a one-frame offset that avoids the singularity at zero age. Smaller $\alpha_k$ means longer memory, following the convention in the fractional-calculus literature summarized by Li's fractional SEIR model. Each summary is a convex combination of real tokens, so it stays close to the distribution the frozen model already understands. This is what makes a training-free version plausible.

Relation to Prior Work

The redundancy side supplies the setting and the strongest competitors. DyCoke merges near-duplicate tokens across adjacent frames during prefilling. It moves pruned tokens into a "recycle bin" cache so that decoding can re-attend to them, and reports a 1.54× speedup on LLaVA-OneVision-7B. Its memory is binary: a token is either kept or parked. We instead make eviction lossy but graded. The parked information is compressed into a constant-size, age-weighted summary rather than stored whole or dropped.

Efficient Video Transformers with Spatial-Temporal Token Selection ranks frames and keeps a top-K budget. It finds that temporal redundancy exceeds spatial redundancy, which motivates compressing along time specifically. Eventful Transformers recomputes only tokens that change. Its reliance on periodic full recomputation to control drift is a separate problem that we do not claim to solve. We also do not claim to settle the density-versus-sparsity tension between ClipBERT and SwinBERT. Our narrower point is that a fixed budget is better spent on graded memory than on hard selection.

The fractional side supplies the kernel and the motivation, but not the system. FRACTAL builds a full state-space architecture that projects history onto polynomials orthogonal under a power-law measure. It reports an average Long Range Arena score of 87.11% against 87.04% for S5. That model must be trained from scratch on one-dimensional sequences and uses fixed orders. We borrow its central premise, power-law measures for history, and apply it as a cache-level operation on a frozen multimodal model.

Neural Fractional Attention Differential Equations combines attention with fractional kernels and beats Transformers on turbulent-flow prediction (RMSE 0.3215 versus 0.4701). However, it stores the full trajectory. Our approach is the opposite trade: bounded memory at the cost of exact recall. The spiking result in Fractional-Order Spiking Neural Network is the closest motivating evidence that power-law memory tolerates temporal gaps. Malti et al.'s tutorial supports the parsimony argument: one fractional order replaces many exponential modes. The same argument is the basis of our strongest ablation below.

As far as the file shows, no prior work turns cache eviction in a video language model into a fractional memory process.

Research Program

The system has four parts. The first is a recent window of full tokens, the last $W$ frames, handled exactly as in the base model.

The second is an eviction rule. By default this is DyCoke-style temporal merging plus age-based eviction, so the comparison isolates what happens to evicted tokens rather than which tokens are evicted.

The third is a set of memory slots. Camera motion breaks the correspondence between a fixed patch position and what it shows. We therefore assign each evicted token to the slot whose current summary key is most similar, falling back to spatial position on a coarse 7×7 grid. We ablate this against pure positional assignment.

The fourth is the kernel bank. We use $K=3$ orders spanning short, medium and long memory, fixed a priori (for example $\alpha \in \{0.9, 0.6, 0.3\}$) so that there is no per-dataset tuning.

Exact power-law sums would cost time linear in video length per update. We therefore approximate each kernel with a short sum of exponentials, about eight terms fitted once offline. This makes each update constant-cost and streaming. Each summary token receives a positional index equal to its kernel-weighted mean frame time, so the model's rotary position encoding sees a plausible timestamp. The total cache is $W$ frames of full tokens plus $49 \times 3$ summary tokens, fixed regardless of video length.

We run the training-free version first on LLaVA-OneVision-7B, the model on which DyCoke reports results. As a second stage, we will lightly fine-tune only the slot-assignment keys and a per-order scalar gain. This tests whether small learned adjustments help and whether the learned effective kernel stays power-law-like.

Baselines

The naive reference is the uncompressed LLaVA-OneVision-7B where it fits in memory, and uniform frame subsampling down to the same total token count where it does not. Subsampling is the "do nothing clever" point: if our method cannot beat evenly spaced frames at equal budget, it has no purpose.

The strongest existing method is DyCoke, which is training-free, uses the same base model, and already addresses temporal redundancy with a recall buffer. On MVBench it reports 57.5 average at retention ratio 0.7, against 58.0 for the unmodified LLaVA-OneVision. For VideoMME and ActivityNet-QA, no reported result was found on Lacuna for DyCoke; we will rerun it with its released configuration.

For fairness, every method gets the same base model, the same frame extraction rate, and the same maximum number of cache entries, with DyCoke's recycle-bin entries counted against the budget. Each method also gets the same small hyperparameter sweep on a held-out split: retention ratio for DyCoke, window size $W$ for all methods. Beating DyCoke means higher accuracy at equal cache size and equal or lower latency. The gain should be concentrated on questions about events more than a few minutes back. We also require recovering at least DyCoke's MVBench retention of the baseline score.

The key ablations replace only the kernel while holding the slot count fixed. The first uses a single exponential moving average with its time constant tuned on the held-out split. The second uses a uniform average over all evicted history, equivalent to $\alpha=0$. The third, most important, is a free mixture of $K$ exponential averages with learned time constants and weights, trained with the same budget as our fine-tuned variant. A further ablation drops the summary tokens entirely, keeping only the recent window and the eviction rule. This measures how much the memory contributes at all.

Why not just…?

Why not just keep more real tokens by subsampling frames more sparsely? Sparse sampling works for coarse retrieval, as ClipBERT shows. But SwinBERT finds fine-grained understanding degrades sharply with fewer frames, with MSRVTT CIDEr dropping from 55.3 at 64 frames to 36.6 at 2. Subsampling also discards events that fall between samples. The uniform-subsampling baseline at equal budget directly tests whether this is good enough. If it matches us across all event distances, the proposal fails.

Why not just use an exponential moving average, which is simpler? This is the most serious threat. Our implementation is itself a sum of exponentials, so the real question is whether the power-law weighting across time constants matters. A single tuned average should win near its own time scale and lose far from it, and the accuracy-versus-distance curve will show this. The free mixture of learned exponentials is the harder test. If it matches the fixed power-law bank, we will report that the benefit comes from multi-scale memory, not from fractional structure. The fractional prior then reduces to a zero-tuning default, which is a weaker but honest result. If the learned mixture converges to power-law-like weights, that supports the claim.

Why not just keep DyCoke's recycle bin, which stores evicted tokens exactly? Exact storage is better for recall only if there is room. Under a hard budget the bin must itself be truncated, and truncation reintroduces irreversible loss. Summary tokens trade exactness for coverage. At a generous budget we expect DyCoke to win. At tight budgets over long videos we expect to win. The budget sweep will locate the crossover rather than assume it.

Why not insert a trained fractional sequence layer like FRACTAL? That would require retraining a multimodal model, and FRACTAL was evaluated only on one-dimensional sequence tasks. Our cache-level approach is cheap enough to test the memory-shape hypothesis first. A positive result would justify the expensive architectural version.

Proposed Experiments

The decisive experiment is a lag-controlled probe built from EPIC-KITCHENS-100, whose timestamped action segments across long egocentric recordings let us generate questions about what the person did at a known distance before the question time. We bin questions by that distance, from under thirty seconds to over twenty minutes, and plot accuracy per bin for every method at three cache budgets.

The prediction is specific. Power-law summaries should match the recent-window-only ablation in the shortest bins, beat the tuned exponential in the longest bins, and beat the uniform average in the middle bins. If the tuned single exponential is statistically indistinguishable from the power-law bank in every bin, the fractional claim is refuted.

Second, we report aggregate accuracy on MVBench, VideoMME (especially its long-video split) and ActivityNet-QA in a streaming protocol. Frames arrive in order, the budget is enforced throughout, and the question is shown only at the end. Here we compare against DyCoke's reported 57.5 on MVBench and our reruns elsewhere.

Third, we measure wall-clock prefill and decode latency and peak memory against DyCoke's reported 1.54× speedup and 1.4× memory reduction. Summary updates must not erase the savings.

Finally, in the fine-tuned variant, we fit the effective age-weighting that the learned exponential mixture converges to. We test whether a power law or a single exponential describes it better. This asks directly whether a model left to its own devices chooses fractional memory.