Gaussian Water-Filling as a Per-Image Visual Token Budget

Vision-language models spend most of their compute on image tokens. Attention cost grows with the square of the sequence length, and a high-resolution image can contribute thousands of tokens. Current methods cut these tokens with hand-tuned rules: keep the tokens that receive the most attention, or keep a fixed 64 tokens per image. The fixed budget is the weak point. A photo of a single cat and a dense receipt get the same number of tokens. The first wastes compute. The second loses the small details a question may depend on.

This note proposes a principled per-image budget taken from classical information theory. Under a covariance constraint, a Gaussian source is the hardest to compress. As a result, the Gaussian rate-distortion function, computed by "reverse water-filling" over eigenvalues, gives a distribution-free upper bound on how much description is needed to preserve a signal with that covariance. We apply this bound to the part of the visual tokens that the text query actually reads. It produces two things at once: a number of tokens to keep, chosen per image and per question, and a linear merging rule for producing those tokens. The method needs no training, and it never forms an $N \times N$ matrix.

Core Claim

The optimal encoder in the vector Gaussian compression problems surveyed here is a linear projection followed by Gaussian noise. Two results establish this. Vector Gaussian CEO Problem Under Logarithmic Loss shows that linear Gaussian test channels achieve the full rate region for distributed noisy observations. Vector Gaussian Successive Refinement With Degraded Side Information shows the same for multi-stage descriptions. The matching rate is set by the eigenvalues of a covariance.

We claim three things for token reduction in vision-language models:

  1. Computing this rate on a query-weighted covariance of visual token features predicts, per image, how many tokens can be removed before answers change.
  2. A merging operator built from the same eigenstructure beats attention-ranked pruning at the same average token count.
  3. The gain concentrates on fine-detail queries, such as text reading, where fixed budgets fail.

The contribution lies in the budget rule and its validation. A new architecture is not the goal.

Concretely, at a chosen layer let $z_i \in \mathbb{R}^r$ be visual token $i$ projected into the low-dimensional subspace spanned by the text tokens' query vectors, and let $a_i$ be its total attention from text tokens. Form the weighted covariance $\Sigma = \sum_i a_i z_i z_i^\top$, with eigenvalues $\lambda_1 \ge \dots \ge \lambda_r$. Reverse water-filling at distortion $D$ picks a water level $\theta$ such that $\sum_j \min(\lambda_j, \theta) = D$. It then assigns rate

$$R(D) = \sum_j \max\!\left(0, \tfrac12 \log \frac{\lambda_j}{\theta}\right).$$

The token budget is $M = |\{j : \lambda_j > \theta\}|$, the number of directions worth describing at all. The summary tokens are built from the corresponding left singular vectors of the matrix with rows $\sqrt{a_i}\, z_i$. That decomposition costs $O(N r^2)$, which is linear in the number of tokens. Because the Gaussian has maximum entropy at fixed covariance, this $M$ is conservative for any feature distribution with the same second moments, measured by squared error in the query-read subspace. This conservatism is the reason to use the Gaussian rather than an arbitrary surrogate.

The argument has a known gap. The softmax is not linear, so the bound covers a linearized proxy of what the language model reads, not the model itself. Whether the predicted $M$ tracks real accuracy cliffs is an empirical question, and it is the central test of this work.

Relation to Prior Work

Two empirical results motivate the project. Which Tokens to Use? compared ten reduction methods and found plain Top-K pruning hard to beat. VisionTrim found that attention-ranked selection drops low-attention details. VisionTrim repairs this with text-guided clustering of the rejected tokens, but still at a fixed budget. Both suggest that the remaining gains lie in deciding how much to keep and in preserving detail, not in better ranking.

FastV supplies the setting. It shows that after the first layers, attention collapses onto prompt tokens, and it prunes a fixed fraction of visual tokens by accumulated attention. We reuse its insertion point and its attention-mass statistic $a_i$. We replace its fixed fraction with a computed budget.

Three information-theoretic papers form the closest bridge, and each one differs from this proposal:

  • Token Statistics Transformer derives linear-time attention from token second-moment statistics via rate reduction. It requires training a new architecture and does not reduce the number of tokens. We keep a pretrained model and use second-moment statistics only to decide what to keep.
  • VTrans uses a variational information bottleneck to prune heads and channels. This leaves the sequence length unchanged.
  • Rate-Distortion Optimization for Transformer Inference codes intermediate tensors for transmission. It reports that bitrate grows with depth (the "Transformer trait").

That last finding appears to contradict FastV's evidence that tokens become redundant in deeper layers. Our decomposition offers a reconciliation that we can test. Per-token description length can grow even while the number of query-relevant directions, $M$, shrinks. We will measure both quantities at each layer.

Scalable Vector Gaussian Information Bottleneck supplies a multi-stage extension: coarse descriptions followed by refinements. We use it only for a secondary experiment that applies refinement at two layers.

Research Program

Measurement. First, we run LLaVA-1.5-7B and LLaVA-NeXT-7B unmodified. For each image-question pair, we find by binary search the oracle minimal $M$: the smallest budget at which the answer, or the token-level log-likelihood of the reference answer, stays within a tolerance. We compare that oracle against the water-filling $M$ at each layer. This step also produces the per-layer curves of $M$ versus per-token rate that bear on the depth contradiction above.

Operator. Second, we build the operator. It inserts after layer 2, as FastV does. It computes $\Sigma$ in the query subspace, takes $M$ from a single global distortion level $D$, and emits $M$ summary tokens.

One design risk deserves its own ablation. Singular-vector combinations have signed weights, and signed mixtures of hidden states may push the language model outside the inputs it saw during training. So we compare two variants. The first uses signed linear summaries. The second, called representative selection, keeps for each retained direction the token with the largest loading and pools the remainder into one package token, in the style of SPViT. Summary tokens receive attention-weighted mean positions.

Sweep. Third, we sweep $D$ to trace an accuracy-versus-average-tokens curve. We compare this curve against fixed-budget methods at matched average token count and matched FLOPs, and we also report wall-clock latency. Per-image budgets produce sequences of varying length, which complicates batching. FastV and SPViT already note that FLOP savings do not automatically become speed savings.

Baselines

No reduction. The do-nothing reference is the full model with 576 tokens for LLaVA-1.5 and 2880 for LLaVA-NeXT. It sets the 100% mark that retained-performance numbers are measured against.

FastV with Top-K. This is FastV's attention-ranked pruning at layer 2 (FastV), run through the same code path as our operator at the same insertion layer. FastV reports up to 45% theoretical FLOP reduction with negligible loss. Following Which Tokens to Use?, plain Top-K is the baseline that complicated methods most often fail to beat.

VisionTrim. This is the strongest existing method. It combines global and local attention selection with text-guided complement tokens, and we run it from the released code (VisionTrim repository) on its own evaluation suite. VisionTrim reports retaining 98.8% of LLaVA-1.5-7B's average performance across 10 benchmarks at 64 of 576 tokens. It also reports 94.0% for LLaVA-NeXT-7B at 160 of 2880 tokens. We first reproduce these numbers on our hardware.

Fairness and the winning bar. Our method chooses its own average budget, so the comparison is made at matched mean token count: 64 and 160 respectively. All methods are training-free, and each gets the same small calibration set for tuning its single hyperparameter (the fraction kept, or $D$). Beating VisionTrim means higher retained performance at the same mean token count on its suite. It also means a clear gain on detail-heavy subsets such as OCR-VQA, without a loss on others.

Key ablations. Four ablations isolate the ingredients:

  • Adaptive budget alone. Water-filling $M$ with Top-K selection tests whether the budget rule matters by itself.
  • Merging alone. Our merging at a fixed $M$ tests the operator without the budget rule.
  • No query weighting. Unweighted covariance, with all $a_i = 1$, tests whether conditioning on the text query is what does the work.
  • Signed summaries versus representative selection. This tests the out-of-distribution risk described above.

Why not just…?

Why not just make Top-K adaptive by keeping tokens until 95% of attention mass is covered? This is the most serious threat. It is simpler, it is also adaptive per image, and it may capture most of the benefit. The two rules differ in one way: attention mass counts how much the text looks at tokens, while water-filling counts how many distinct directions those looks occupy. Twenty near-duplicate high-attention tokens raise the mass but add only one direction. We will run the mass-threshold rule at matched mean budget. If it matches our method, both overall and on OCR-VQA, the Gaussian budget adds nothing, and we will report that.

Why not just take a low-rank SVD of the visual tokens, with no information theory? Without query weighting and a water level, a truncated SVD answers "how many directions does the image have." The relevant question is "how many directions does this question read," and an SVD offers no principled stopping rank. The unweighted ablation, plus a fixed-rank SVD at matched budget, directly measures what the Gaussian rate adds over plain low-rank approximation.

Why not just use VisionTrim's text-guided complement at a smaller fixed budget? VisionTrim already handles query-dependent detail. But its own 1-token stress test, which retains 82.8% of performance, shows that fixed budgets degrade gradually on average while hiding failures on individual images. Our per-image oracle analysis tests whether the hard images are predictable from $\Sigma$. If they are not, adaptivity cannot help, and VisionTrim remains the better choice.

Why not just train a linear-attention model such as ToST? Token Statistics Transformer gives up accuracy: ToST-S reaches 77.9% versus 79.8% for ViT-S on ImageNet-1K. It also requires retraining, which is not practical for pretrained 7B vision-language models. The two approaches are complementary, not competing.

Decisive Checks

The first check is whether the water-filling $M$ predicts the oracle minimal $M$. We will measure rank correlation across images on OCR-VQA, MME, MMMU and Nocaps, the image benchmarks used by FastV. A weak correlation, or one no better than that of the attention-mass threshold, falsifies the core claim, regardless of average accuracy.

The second check is retained performance at matched mean budget against reproduced VisionTrim on its suite, for both LLaVA models. We expect the largest gap on OCR-VQA.

The third check is video. We run Video-LLaVA on TGIF and MSRVTT, where VisionTrim reports 98.0% retained performance at 136 of 2048 tokens. Frames vary widely in redundancy, so adaptive budgets should matter most there.

The fourth check covers the per-layer curves of query-relevant rank and per-token rate. We predict that rank falls with depth while per-token rate rises. This reconciles the depth-dependent bitrate growth reported in Rate-Distortion Optimization for Transformer Inference with FastV's evidence of redundancy. If both quantities rise together, the reconciliation fails, and that counts against the framing.

Finally, wall-clock latency at batch sizes 1 and 8 will show whether variable-length budgets keep their FLOP advantage in practice.