Vision transformers compare every image patch with every other, so their cost grows with the square of the number of patches. The usual fix is to drop or merge patches the network considers redundant. These methods, such as Top-K pruning, Token Merging and adaptive patch sizing, work well for whole-image classification. They fail quietly on images whose meaning lives in thin structures: roads, nerve canals, neuron membranes, fingers. A one-patch-wide road resembles its background on most sides, so merging it away saves a lot of tokens and costs almost nothing under any similarity score. Classical segmentation has a name for this failure. Length and area penalties have a shrinking bias: they cut away thin, elongated parts because those parts contribute little to the total. The fix there was to penalize curvature instead, because curvature is scale-invariant and does not reward shrinking.
We propose to carry that fix into token reduction. Squared curvature can be counted on a grid with a simple local test from integral geometry. A collinear triple of positions $(p-\Delta, p, p+\Delta)$ "fires" when the center disagrees with both ends. We apply this test to transformer token features and use the firing score to decide which tokens must never be merged or coarsened. The test is cheap: a few shifted comparisons per token, done in parallel on the GPU. It also fires only on thin parts and tight bends, not on every edge. The claim is that, at an equal token budget, this criterion keeps thin-structure segmentation intact where entropy-, attention- and edge-based criteria do not.
Core Claim
Efficient Squared Curvature showed that the number of firing collinear triple cliques (patterns 0-1-0 or 1-0-1) is proportional to a boundary's squared curvature. It used this count as an energy term in binary 2D segmentation and optimized it with trust-region graph cuts in up to 110 seconds per image. We use the same counting identity for a different purpose. We do not optimize a labeling. We score tokens to protect a reduction step.
The geometric property that matters is selectivity. Take a structure narrower than $\Delta$. Every point on it fires in the direction across the structure. At the edge of a thick region, one neighbor lies inside and one outside, so nothing fires. Plain edge strength, by contrast, flags every boundary equally. The test also fires at sharp corners of thick regions, at a rate tied to curvature. So it puts the protection budget on the two places merging destroys: thin parts and tight turns.
On features, we use a soft version of the test. Let $x_p$ be the token feature (attention key) at grid position $p$, $d(\cdot,\cdot)$ cosine distance, and $\mathcal{U}$ a set of 4 directions in 2D or 13 in 3D. The thinness score is
$$s_\Delta(p) = \max_{u \in \mathcal{U}} \Big[ \min\big(d(x_p, x_{p-\Delta u}),\, d(x_p, x_{p+\Delta u})\big) - d(x_{p-\Delta u}, x_{p+\Delta u}) \Big]_+ .$$
The score is large only when $p$ differs from both collinear neighbors and those neighbors agree with each other, which is the soft 0-1-0 pattern. We compute it at $\Delta \in \{1, 2\}$ tokens and take the maximum. The cost is $O(N|\mathcal{U}|)$ for $N$ tokens, negligible next to attention.
Research Program
We insert the score at two points that existing methods leave geometry-blind.
The first insertion point is inside the network. Token Merging, from Token Merging: Your ViT But Faster, merges the most similar tokens through bipartite matching on keys. Agglomerative Token Clustering merges bottom-up with linkage on key distances. Both use only feature similarity. Before each merge step we exclude the top-$\rho$ fraction of tokens by $s_\Delta$ from being merged. The rest of each algorithm stays unchanged, including Token Merging's proportional attention. For dense prediction, merged tokens are copied back to their source positions before the decoder.
The second insertion point is at the input. Adaptive patch sizing, from Accelerating Vision Transformers with Adaptive Patch Sizes, gives large patches to low-entropy regions. A faint, low-contrast vessel can have low entropy. We replace entropy with $s_\Delta$ computed on raw intensities over small pooled blocks. A block keeps fine patches if any sub-block fires. Everything else stays as in that paper: sequence packing, block-diagonal masks and zero-initialized fusion layers.
We also test a training-time variant. Here the network learns protection scores, and we add an auxiliary loss so that a token's predicted keep-probability tracks the triple-clique firing on the ground-truth mask. The firing pattern is computable from the label map in linear time. This tells us whether a learned criterion beats the fixed one.
The work sits relative to three groups of prior papers. Token-reduction papers supply the methods we modify: SG-Former learns significance maps, and TCFormer clusters tokens by density peaks. Both select tokens by learned salience rather than geometry. TCFormer's +6.2% AP gain on hand keypoints is evidence that spatially guided allocation matters for small parts.
Topology-preserving losses address a related but different problem. clDice in the mandibular canal study and persistence matching in TopoMatch fix connectivity errors through the training objective, but they leave compute unchanged. TopoMatch also costs $O(n^3)$ in pixels. We treat these losses as complementary and use clDice for all methods.
MapVR adds differentiable rasterization to map vectorization. It is the closest system that combines thin-geometry supervision with a transformer, but it does not touch backbone cost. We do not compete with it directly.
Baselines
The naive reference is the full-token model: a ViT-B segmenter, with a UNETR-style ViT encoder for 3D, trained without reduction. It sets the accuracy ceiling and the throughput floor.
The strongest in-network competitor is Agglomerative Token Clustering. The file reports that it beats Token Merging by 4 mAP on COCO detection at a 50% keep rate and by 9.6 points on NABirds at 25% keep, so it is the best geometry-blind merger on record. We run it, Token Merging, and Top-K/EViT pruning on the same trained full-token checkpoint, at identical keep rates (75%, 50%, 25%). Top-K/EViT is included because Which Tokens to Use? found it matches or beats learned schemes. Every method also gets the same short fine-tuning budget, and each method's hyperparameters get the same number of tuning trials.
The input-level competitor is entropy-based adaptive patching, at matched token counts. The file reports 28–32% token reduction on ADE20K without degradation, but no absolute mIoU.
The external accuracy reference on the canal task is Swin-UNetR with clDice and DRC: Dice 0.844, HD95 2.947 mm, IoU 0.731 (mandibular canal study). On Massachusetts Roads and CREMI we found no reported numeric Dice or Betti value on Lacuna, so we reproduce the full-token model with clDice as the reference.
The key ablation replaces $s_\Delta$ with edge strength, measured as feature-gradient magnitude at the same protection fraction $\rho$. A second ablation uses a random protected set of the same size. Beating these means protected-token reduction loses at most a third as much as the best baseline, relative to full-token accuracy. The losses are measured on clDice and Betti error at 25% keep rate, with throughput within 5% of Token Merging's.
Why not just…?
Why not just use Top-K pruning or Token Merging at a gentler rate? At gentle rates they lose little. That is exactly what the classification results show: Token Merging gives 2x throughput for a 0.2–0.3% drop. But the Token Merging paper itself leaves dense prediction open. We report full accuracy-versus-throughput curves, so a gentler-rate baseline is just another point on its curve. Our method wins only if its curve dominates on thin-structure metrics.
Why not just protect tokens by edge strength, a far simpler signal? This is the real threat, and the edge-strength ablation exists to settle it. Our prediction is specific. On Massachusetts Roads, where almost all foreground is thin, edge protection should come close to ours. On ADE20K and COCO-WholeBody, edges spend the budget on both sides of thick boundaries, and the triple-clique score should win at tight budgets. If edge protection matches ours everywhere, we will report that the curvature framing adds nothing beyond "protect boundaries."
Why not just train with clDice or TopoMatch and keep standard merging? Losses shape what the network represents. They cannot recover a road that was merged into background before the decoder ever saw it. We train every baseline with clDice, so any remaining gap comes from the reduction step itself. TopoMatch's cubic cost and 2D-only implementation rule it out for the 3D canal.
Proposed Experiments
On Massachusetts Roads and CREMI we measure Dice, clDice, Betti-number error and images per second at three keep rates. On the 3D mandibular canal dataset we measure Dice, HD95 and peak memory, paying particular attention to the faded head and tail segments where the reference model degrades. ADE20K mIoU and COCO-WholeBody hand AP test whether the gains carry over to natural images. ImageNet-1K Top-1 checks that classification does not suffer.
Three checks decide the outcome. First, the triple-clique protected set must beat edge-strength protection at 25% keep on at least two of ADE20K, COCO-WholeBody and CREMI. Second, the gain over Agglomerative Token Clustering must grow as the budget tightens, which is the signature of a shrinking-bias fix. Third, the per-structure-width breakdown on Roads must show that the recovered errors sit on segments narrower than $\Delta$ tokens, where the firing test predicts them. If the gains appear at all widths uniformly, the mechanism is not what we claim.
References
- Efficient Squared Curvature
- Token Merging: Your ViT But Faster
- Agglomerative Token Clustering
- Accelerating Vision Transformers with Adaptive Patch Sizes
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer
- Automatic segmentation of mandibular canal using transformer based neural networks
- Topologically Faithful Image Segmentation via Induced Matching of Persistence Barcodes
- Online Map Vectorization for Autonomous Driving: A Rasterization Perspective
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers