Local Suppression for Visual Token Pruning in Diagram Reasoning

Vision-language models read an image as thousands of small patch "tokens" before reasoning about it, and that sequence length dominates their inference cost. A popular way to cut it is to borrow non-maximum suppression (NMS), the clean-up step object detectors use to delete duplicate boxes. A token is dropped if a higher-scoring token looks too similar to it. This works on photos and conversational questions. Diagrams are a different case. Two tick marks on an axis, two vertex labels, or two parallel line segments look almost identical in feature space, yet each is a separate fact the model needs for its reasoning. Similarity-based suppression treats them as duplicates and deletes all but one. Box NMS never makes this mistake, because a box can only suppress boxes it physically overlaps.

We propose to restore that locality to token suppression. A token may suppress another only if the two are both similar and spatial neighbors, so redundancy is removed within one visual primitive but never across distinct ones. The same restriction also replaces an all-pairs similarity computation, which grows quadratically with token count, with a neighborhood computation that grows linearly. This matters for the very high-resolution figures where pruning is most needed. The claim is testable. At equal token budgets, locally gated suppression should keep more of the annotated geometric and chart primitives, recover more accuracy on mathematical-diagram benchmarks than global similarity suppression, and spend less time in the suppression step itself.

Where this sits

The closest work is TokenNMS from Decoupling Spatial and Semantic Token Compression for Vision-Language Model Acceleration. It is a training-free method that first spreads kept tokens across the image grid, then suppresses tokens whose features are highly similar to higher-scoring ones. Its suppression step is global in feature space. Our method changes exactly that step. TokenNMS was evaluated on general VQA and captioning, not on diagram reasoning, and it does not analyze how its pairwise similarity step scales.

Prune Redundancy, Preserve Essence (PruneSID) prunes by semantic clustering plus importance. It reports cutting visual tokens "by over 90% while maintaining nearly full performance" and a "7.8x speedup in image token processing." It was not tested on fine-grained symbolic content either. Both methods supply the technique we modify; neither is a reasoning-oriented study.

Three lines of evidence motivate the worry that these methods damage reasoning:

  • Open Eyes, Then Reason reports that GPT-4o has a 70% error rate identifying geometric entities, even with all tokens kept, so primitives are already fragile.
  • Plot Twist shows that models do worse on low-level chart readout (axes, point values) than on high-level questions, which suggests that accuracy can hide lost detail.
  • MATH-Vision finds that the dominant failure comes after recognition, in applying the right operations.

This last finding constrains us. Better pruning can only stop compression from adding perception errors on top of the existing reasoning gap; it will not close that gap. We frame the contribution accordingly.

From the detection side, two results supply the analogy and the warning. Fast NMS in YOLACT and the soft score decay of Matrix NMS in SOLOv2 both parallelize suppression, and both are known to over-suppress in crowded scenes. Hashing-based work on crowded detection, Hashing-based Non-Maximum Suppression for Crowded Object Detection, addresses cost by restricting comparisons to local buckets. We transfer that "compare only nearby candidates" idea to tokens.

Core Claim

Each token $i$ has an importance score $s_i$ (the same query-aware score TokenNMS uses), a grid position $p_i$, and a feature vector $f_i$. We decay each token's score as follows:

$$\tilde s_i = s_i \prod_{j:\, s_j > s_i,\ \|p_i - p_j\|_\infty \le r} \big(1 - \sigma(\cos(f_i,f_j) - \tau)\big)$$

Here $r$ is a neighborhood radius in grid cells, $\tau$ is a similarity threshold, and $\sigma$ is a sharp sigmoid. The soft product follows Matrix NMS so that it runs in one parallel step. We then keep the top-$K$ tokens by $\tilde s_i$. Setting $r=\infty$ recovers global similarity suppression, which makes the key ingredient directly ablatable.

The cost is $O(N r^2)$ instead of $O(N^2)$, since each token compares only against its $(2r+1)^2$ neighbors. We predict that suppression-step latency stays roughly flat per token as resolution grows, while the global variant's cost grows linearly per token.

Research Program

We first build a diagnostic. It is a programmatic generator of geometry figures, function plots, and bar/line charts in which every primitive (vertex, label, tick, segment, bar) has a known pixel mask. On these figures we measure "primitive survival": the fraction of primitives that still have at least one kept token covering them after pruning. We also report survival for primitives that have a near-identical twin elsewhere in the image, which is the case we argue global suppression breaks. On ViEBench, whose expert-annotated evidence regions serve the same role on real images, we measure the fraction of evidence-region tokens retained.

Second, we implement the locally gated suppression as a drop-in replacement for the second stage of TokenNMS. It runs inside one open high-resolution VLM backbone, and the whole computation stays on the GPU as a banded neighborhood operation.

Third, we run reasoning evaluations on MathVista, MATH-Vision, and MathVerse (as used in Open Eyes, Then Reason). As a scientific-figure check we use VeriSciQA. All runs use token budgets of roughly 25%, 11%, and 5%, and we report accuracy, primitive survival, and wall-clock latency split into suppression time and decoding time.

Finally, a scaling and stress test feeds figures of increasing resolution, plus synthetic images of many visually diverse small marks. These are chosen to resist suppression, in the spirit of the Overload latency attack. The test checks whether the suppression step itself becomes the bottleneck.

Baselines

The naive reference points come first:

  • No compression. The full-token backbone sets the accuracy ceiling and the latency floor to beat.
  • Uniform reduction at matched budget. Image downsampling and fixed-grid patch dropping are what a practitioner would try first.

The strongest existing methods are PruneSID (Prune Redundancy, Preserve Essence) and TokenNMS (Decoupling Spatial and Semantic Token Compression). PruneSID's reported figures (over 90% fewer tokens, 7.8x faster image-token processing) are on general benchmarks. On reasoning benchmarks there is no reported result found on Lacuna for either PruneSID or TokenNMS on MathVista, MATH-Vision, or MathVerse, so we will run both ourselves. For context, the only full-model MathVista number in the file is GPT-4V at 49.9% overall accuracy (MathVista). That is a different model, so it is not a target, but it calibrates how hard the benchmark is.

All methods use the same backbone, the same importance scorer where applicable, and the same token budgets. Each method gets an equal hyperparameter search (thresholds, cluster counts, $r$, $\tau$) on a held-out development split drawn from the benchmarks' training or validation material, never the test items. Latency is measured on the same GPU with batch size one.

Two ablations isolate the key ingredient:

  • Global versus local gating. Our code with $r=\infty$ isolates the effect of locality, holding everything else fixed.
  • Local average merging. Pooling similar neighboring tokens instead of suppressing them tests whether keep-the-best selection matters at all, or whether any local reduction suffices.

We count a win under three conditions. At the 11% and 5% budgets, the method must exceed TokenNMS and PruneSID on MATH-Vision and MathVerse accuracy by a margin outside the bootstrap confidence interval. It must show higher twin-primitive survival. And its suppression-step latency must be no higher than global suppression's.

Why not just…?

Why not just keep more tokens? If accuracy loss vanishes at a mild budget, the problem is moot. We address this with accuracy-versus-latency curves rather than single points. The method is only useful if it shifts that curve, giving the same accuracy at fewer tokens. If all pruning methods converge once about 25% of tokens are kept and differ only at 5%, we will report that the benefit is confined to aggressive regimes.

Why not just downsample the image? Open Eyes, Then Reason shows that small primitives already fail at full resolution, and TokenNMS's authors report that naive pruning causes spatial bias and context loss. Downsampling removes exactly the thin lines and small labels in our survival diagnostic. The matched-budget downsampling baseline will show this directly, or embarrass us if it does not.

Why not just tune TokenNMS's similarity threshold higher? This is the most serious threat. A strict threshold suppresses fewer twins. It also stops removing real redundancy, so at a fixed budget the top-$K$ step must cut elsewhere. The $r=\infty$ ablation, tuned with the same budget, settles the question. If a well-tuned global threshold matches local gating on accuracy, the remaining contribution is only the latency scaling, and we will say so.

Why not just transcribe the figure to text and skip visual tokens? Synthesize Step-by-Step does this for charts. However, GeoSQA shows that even perfect human descriptions do not solve the reasoning. Reliable diagram-to-text parsing also presupposes the primitive recognition that current models lack. This alternative answers a different question from ours, and it does not remove the need to read the figure cheaply.

Why not just merge neighboring tokens instead of suppressing them? Local merging is close to our idea and simpler. It is included as an ablation. If it matches suppression, the finding to report is "locality, not selection," which is still the claim's core.

Decisive Checks

The proposal stands or falls on three measurements. First, on synthetic diagrams with twin primitives, global suppression must lose measurably more twin primitives than local gating at the same budget. If it does not, the motivating failure does not exist. Second, that survival gap must translate into accuracy gaps on MATH-Vision and MathVerse at the 5–11% budgets, beyond what threshold tuning of TokenNMS recovers. Third, suppression time for global similarity must grow faster with resolution than for local gating, and must become a visible fraction of prefill time on the highest-resolution VeriSciQA figures. If only the third holds, this is an efficiency paper. If none holds, similarity-based token NMS is safe for diagram reasoning, which is itself a useful negative result for the compression literature.