Spending Attention Where Weak Seeds Are Unsure

Weakly supervised segmentation trains a pixel-level segmenter from image-level tags alone ("this image contains a dog"). It does this by first producing seeds, coarse heat maps of where each class appears, and then expanding them into dense pseudo-labels. Recent methods produce these seeds by comparing each image patch of a frozen CLIP model (a vision-language model trained to match images with captions) against a class representation. Two problems meet here. Seeds miss thin and small objects unless the image is processed at high resolution. But a vision transformer's self-attention cost grows with the square of the patch count, so doubling resolution multiplies attention cost by sixteen. The usual fix for that cost is token pruning: drop the patches the model pays least attention to. In weak supervision that fix is dangerous. The patches that look least relevant are often the non-discriminative object parts the seed has not yet found, and refinement cannot recover a region that has no seed.

We propose to make the high-resolution seed pass cheaper by merging tokens, never dropping them. The merge decision comes from uncertainty measured against vision-space class prototypes, not from relevance to text. A cheap low-resolution pass labels each region as confidently one class, confidently background, or ambiguous. The high-resolution pass then keeps full patch density only in ambiguous regions and coarsens confident ones, and every merged token is copied back to its pixels at the output. The claim in two sentences: at a fixed compute budget, where a seed generator spends its tokens determines seed completeness. Allocating tokens by text similarity is systematically wrong because of the gap between text and image embeddings, so it should be replaced by allocation based on vision-prototype uncertainty.

Core Claim and Why It Is Not Already Done

Three pieces of prior work set up the question. VLTP prunes SAM's vision backbone using cross-attention to a language-derived segmentation token. It reports roughly 25% fewer backbone GFLOPs with no loss, and 40% fewer with a 1% mIoU drop, but it needed a Token Reactivation module to recover boundaries. It is fully supervised and assumes the language guidance is accurate. FastV and AgilePruner prune visual tokens by attention in vision-language chat models. AgilePruner also shows that the obvious escape, keeping diverse tokens instead, raises hallucination (CHAIR $C_S$ rises from 51.0 unpruned to 57.4+). None of these works produces pseudo-labels, and all of them discard tokens permanently.

On the seed side, Vision Prototype Learning (VPL) shows that CLIP text embeddings lack the texture and lighting detail needed for dense localization. It learns class prototypes directly in visual feature space, using text only as an initial guide. This improves CLIP-ES seeds by 5.5% and reaches 78.5% mIoU on PASCAL VOC val and 49.2% on COCO. VPL runs full quadratic attention at uniform resolution. Its authors note that the prototypes are static.

Our contribution is to treat VPL's finding as a claim about allocation. If text similarity misjudges non-discriminative parts, then any compute scheme driven by text relevance, as VLTP's is, will cut exactly those parts. Vision-space prototypes should give a better signal of which regions are still undecided.

SG-Former and TCFormer also vary token density across an image, but they give density to salient regions and are trained backbones under full supervision. We give density to uncertain regions, which usually means object extents and boundaries rather than salient cores, inside a frozen CLIP encoder with no retraining.

Research Program

The system has two passes over a frozen CLIP ViT-B/16 image encoder.

Coarse pass. It runs at standard resolution, for example 224 pixels on the short side. We compute per-patch class posteriors against VPL vision prototypes, which we reproduce using the paper's KL-guided training. We also compute a per-patch ambiguity score, $u = 1 - (p_{(1)} - p_{(2)})$, where $p_{(1)}$ and $p_{(2)}$ are the largest and second-largest class posteriors (background included) for that patch.

Fine pass. It runs at 2–4× resolution. We upsample the ambiguity map to the fine patch grid. Each $2\times2$ or $4\times4$ block whose ambiguity falls below a threshold $\tau$, and whose coarse label agrees across the block, is replaced by one averaged token. That token's attention logit carries a log-count bias so that a merged region is not under-weighted. $\tau$ is set per image so that the total token count meets a fixed budget $B$, which keeps sequence lengths constant and batchable and addresses the hardware problem of variable-length pruning. After the final layer, merged tokens are broadcast back to their constituent patches, and seeds are computed exactly as in CLIP-ES plus VPL.

Nothing is discarded. A region the coarse pass confidently called background is still processed, only at coarse resolution. This matters because confident false negatives, whole objects missed by the seed, are the failure that iterative affinity learning cannot repair.

The analysis measures what we call the irreversibility rate: the fraction of ground-truth foreground pixels that fall in tokens removed by a pruner, or coarsened by our merger. We report it per layer and split it into discriminative-core and non-discriminative-extent pixels. We define the split with the pixels activated by the plain CLIP-ES seed versus the ground-truth object pixels it misses.

This diagnostic tests the cross-modal hypothesis directly. If text-relevance allocation hits extent pixels far more often than vision-prototype allocation does, the modality gap is shown to matter for efficiency, and not only for seed accuracy.

A secondary analysis addresses the disagreement between FFR, which says attention over-smooths boundaries like a low-pass filter, and MoRe, which says the problem is spurious correlation with background. Our scheme keeps full-resolution tokens at ambiguous boundaries and coarsens homogeneous interiors, so we can ask which kind of seed error it reduces. We measure boundary blur via boundary F-score against false-positive background islands, counted as connected components outside the ground truth. We treat this as a readout, not a design goal.

Baselines

The naive reference is CLIP-ES with VPL prototypes run at standard resolution under full attention. It is the cheapest option and shows how much resolution matters at all.

The strongest reference for accuracy is the same pipeline at our fine resolution under full attention. It is the quality ceiling our method tries to match at lower cost. VPL reports 78.5% VOC val and 49.2% COCO val mIoU for final segmentation, and a 5.5% seed gain over CLIP-ES. No reported result was found on Lacuna for the GFLOPs of CLIP-based seed generation at any resolution, so we measure both ends ourselves.

The compute-matched uniform baseline runs full attention at whatever single resolution uses the same FLOPs as our budget $B$. This is the comparison that decides whether allocation helps, rather than resolution alone.

The strongest existing efficiency method is a VLTP-style allocator. Tokens are scored by cross-attention to the CLIP text embedding of each present class. The lowest-scoring tokens are pruned after early layers, and the pruned tokens are reactivated at the output, as in VLTP. We also run a FastV-style variant that scores by attention received and applies no reactivation.

Our method has two ingredients, and three ablations separate them. The first swaps our vision-prototype ambiguity score for text-similarity ambiguity while keeping merging. The second keeps our score but prunes instead of merging. The third uses random block merging at the same budget.

All methods share the frozen encoder, the same VPL prototypes, the same budget $B$ in measured GFLOPs, and the same threshold search on a 500-image held-out subset of the training set. Pseudo-labels from each method then train an identical DeepLab-style segmenter with the same schedule.

Beating the baselines means two things. At no more than 40% of the fine-resolution full-attention GFLOPs, our seeds must come within 0.5 mIoU of that ceiling. And at equal GFLOPs, they must beat both the compute-matched uniform baseline and the VLTP-style allocator by at least 1.5 seed mIoU on COCO, where small, cluttered objects make allocation matter most.

Why not just…?

Why not just run the seed pass at full resolution, since pseudo-labels are computed once offline? This is the most serious objection. If full-resolution attention is affordable, efficiency gains alone are a minor engineering contribution. Our answer is that the claim is about accuracy at equal compute, not only about cost. At a given budget, a well-allocated high-resolution pass should beat uniform resolution, and that advantage carries over to settings where compute is genuinely binding: multi-scale test-time augmentation, 4× resolution on COCO, and large corpora. If the compute-matched uniform baseline ties our method, we will report that allocation does not matter for seeds, and the efficiency story shrinks to a speedup.

Why not just prune tokens with text guidance and reactivate them, as VLTP does? Reactivation restores tokens only at mask decoding. Pruned tokens never took part in the middle layers where context spreads into object extents, so a missed leg or tail stays unrepresented. VPL gives the reason to expect text scoring to fail on exactly those parts. The irreversibility-rate split and the text-versus-vision ablation test this directly. If text allocation matches ours, the modality-gap argument is wrong, and we will say so.

Why not just downsample, or use a linear-attention backbone such as QT-ViT? Downsampling is our compute-matched uniform baseline, and it is expected to lose small-object recall, which we report by object-size bin. Linear attention needs the vision encoder retrained. That would break CLIP's pretrained image-text alignment, on which both seeds and VPL prototypes depend. QT-ViT was evaluated only on fully supervised unimodal tasks, and whether its kernel preserves cross-modal localization is an open question that is too large to fold into this project.

Why not just refine seeds with SAM, as Foundation Model Assisted WSSS does? SAM sharpens boundaries of regions that already have seeds. It reaches 82.6% VOC and 55.4% COCO with a Swin-L segmenter, but it adds a backbone that VLTP measures at over 90% of system latency. SAM refinement is complementary to our method rather than a competitor. We will report it on top of both our seeds and the uniform baseline to check that our gains survive SAM refinement.

Proposed Experiments and Decisive Checks

Experiments use PASCAL VOC 2012, with the 10,582-image augmented training set for pseudo-labels and val/test for segmentation, and MS COCO 2014 with 80 classes plus background. We report seed and pseudo-label mIoU on the training set, final segmentation mIoU, seed-stage GFLOPs and wall-clock time on one GPU, recall by object-size bin, and the irreversibility rate split into core and extent pixels. We sweep budget $B$ from 20% to 100% of full fine-resolution cost to trace accuracy-compute curves for every method.

Three results would settle the proposal. First, on COCO our curve should lie above the compute-matched uniform curve across the middle of the budget range. Second, the text-scored allocator should show a clearly higher extent-pixel irreversibility rate than the vision-prototype allocator at the same budget. Third, the prune variant of our own score should lose more small-object recall than the merge variant.

If the first result fails, allocation does not matter for weak seeds. If the second fails, the modality gap does not explain which tokens are safe to spend less on. If the third fails, merging buys nothing over pruning plus reactivation. Each outcome is informative, and together they turn the "prune what looks irrelevant" habit from efficient transformers into a testable question for weak supervision.