Contrast-First Token Admission for Single-Pass Weakly Supervised Localization

Vision Transformers can learn where objects are from image-level labels alone. This task is called weakly supervised object localization: the model sees only "this is a cardinal," never a box, and must still draw one. The best current methods pay for that ability twice. Every image patch attends to every other patch, so cost grows with the square of the patch count. The leading method then runs a second attention pass to clean up attention that has spread onto the background. We propose a cheaper route. A nearly free, label-blind measure of local visual contrast (how much a patch's color and edges differ from its neighbors) decides early in the network which patches deserve to enter the expensive attention layers. During training, a self-paced schedule hands that decision over to the network's own learned class evidence, one image at a time and only once the network classifies that image confidently. The output is a localization map from a single forward pass that attends over roughly a third of the patches.

The bet is narrower than "saliency helps" or "pruning saves compute," and both of those claims are already known. Our claim is about how pruning interacts with training. If a network prunes patches with its own class evidence from the start, it should make the localization gap worse. A young network's evidence sits on the most discriminative parts, such as a bird's head. Pruning by that evidence permanently discards the body and tail, which are exactly the regions weak supervision already neglects. Contrast does not depend on the class, so it is not biased toward discriminative parts, and it can protect the whole object's extent while the network is immature. Contrast is also fooled by cluttered backgrounds and fails on camouflaged objects. For that reason it should give way to learned semantics as the network becomes reliable, rather than be fused with it at a fixed weight.

What exists and how this differs

The closest prior work falls into three groups. Each covers two of our ingredients but not the third.

The first group pairs saliency with weak supervision but leaves cost and pacing aside. Bridging Saliency Detection to Weakly Supervised Object Detection Based on Self-Paced Curriculum Learning uses fixed Region Contrast saliency to pick "easy" training images and then adds harder ones by loss ranking. Its curriculum runs over which images are trained on, inside an SVM-and-proposals pipeline. We schedule something different: which signal controls token admission inside one network. Saliency Guided Self-Attention Network for Weakly and Semi-Supervised Semantic Segmentation multiplies an off-the-shelf saliency map into self-attention after the full pairwise attention has been computed. That stops leakage but saves nothing, and it trusts saliency equally on every image. Both papers motivate the proposal, since they show contrast carries real localization signal. SGAN also constrains it: its documented failure cases, such as a wooden table on a wooden floor and high-contrast clutter, are why contrast must not keep the final say.

The second group fixes Transformer leakage at extra cost. Re-Attention Transformer for Weakly Supervised Object Localization (TRT) computes full attention, thresholds the class-to-patch scores with a dataset-specific cutoff, and re-runs attention over the surviving foreground tokens. Its pruning signal is semantic only, and pruning happens after a full pass rather than instead of one. ViTOL drops peak patches during training to push attention toward secondary parts. LCTR restores local continuity with a full patch-relation matrix. None of the three uses a bottom-up cue, and none reduces inference cost. They supply our strongest baselines and the class-attention scoring we reuse.

The third group targets efficiency outside weak supervision. DEFA prunes about 84% of deformable-attention sampling points, but it relies on fully supervised detectors and box feedback. Hard-Coded Gaussian Attention for Neural Machine Translation shows that fixed local attention can match learned attention (30.3 vs. 30.0 BLEU on IWSLT16 En-De) yet fails on long-range dependencies. That failure constrains our design: we keep learned global attention among admitted tokens and spend the efficiency budget on choosing which tokens enter, not on locking attention to a fixed local shape.

Core Mechanism

We start from a DeiT-Base backbone, the same family TRT reports. The first $L_0$ layers, with $L_0 = 3$ by default, run full attention, which is cheap at that depth relative to the whole network. After layer $L_0$, each patch $i$ receives an admission score

$$s_i = \alpha(x)\, c_i + \big(1-\alpha(x)\big)\, a_i ,$$

The terms in this score are defined as follows. $c_i$ is the patch's rank-normalized bottom-up contrast, computed from raw pixels as the center-surround difference in mean color and gradient energy against its eight neighbors, at linear cost in the number of patches. $a_i$ is the rank-normalized attention from the class token to patch $i$ at layer $L_0$. $\alpha(x) \in [0,1]$ is a per-image mixing weight.

The top $K$ patches by $s_i$ continue through the remaining layers. All other patches are averaged, weighted by score, into a single context token, so the classifier still sees a summary of the background. The deep layers therefore cost $O(K^2)$ instead of $O(N^2)$, where $N$ is the full patch count.

The localization map is the TS-CAM-style product of class-token attention and semantic maps over the admitted patches. Pruned positions are set to zero. This deliberate choice means a wrong pruning decision is visible in the localization score rather than hidden.

The curriculum lives in $\alpha(x)$. Following the self-paced logic of Self-Paced Learning for Latent Variable Models, we set

$$\alpha(x) = \sigma\!\left(\frac{\ell(x) - \lambda_t}{\tau}\right).$$

Here $\ell(x)$ is the image's current classification loss, tracked as a running average across epochs. $\lambda_t$ is a pace threshold that falls over training, and $\tau$ is a temperature.

Images the network already classifies confidently use mostly learned semantic admission. Poorly classified images stay on contrast admission. As $\lambda_t$ falls, more images hand over. The budget $K$ also anneals, from $N$ down to the target, for example $N/3$, over the first part of training. This avoids the instability ViTOL's dropping can cause when it is applied from the first epoch.

We choose this form over a global epoch-based schedule for a specific reason. The curriculum literature's main complaint is hand-designed difficulty proxies whose pace does not match the model's actual learning. Tying the handover to each image's own loss removes most of that mismatch, and it leaves two pace parameters, $\lambda_t$ and $\tau$, that we will report sensitivity sweeps for.

Research Program

The work has three parts.

The first part builds the admission module and measures, on the CUB-200-2011 and ImageNet-1K training sets, what fraction of ground-truth box area each scoring rule keeps at a fixed budget $K$. We measure this coverage at several training checkpoints. It tests the central mechanism directly, before any localization number is computed. We expect semantic-only admission to keep the discriminative part and drop the rest early in training, and contrast-only admission to cover the object but also admit cluttered background. If early semantic-only coverage is not clearly worse than contrast coverage, the main hypothesis is wrong, and we will say so.

The second part trains the full model and its ablations end to end and evaluates localization and cost.

The third part is a failure analysis. We split each test set by how much the contrast ranking and the final learned ranking disagree within the box. This isolates the cluttered-background and camouflage cases, so we can check whether the handover actually rescues the images where contrast misleads.

Baselines

The naive reference point is a plain DeiT-Base fine-tuned for classification, with a TS-CAM-style map from full attention and no pruning. On CUB, TRT's paper implies about 90.8% Gt-Known for TS-CAM, which is TRT's reported 3.3-point gain subtracted from its 94.1%. We also include the parameter-free Center Gaussian blob. Evaluating Weakly Supervised Object Localization Methods Right reports that it beat several 2018–2019 methods on CUB, but no numeric value for it is reported in our file, so we will compute it ourselves.

The strongest existing competitor is TRT. It reports 94.1% Gt-Known on CUB-200-2011 and 70.7% Gt-Known on ImageNet-1K with DeiT-Base. It is the right target because it attacks the same leakage problem with token selection, and because its two-pass design is exactly the cost we claim to remove. Under the threshold-free MaxBoxAcc-V2 metric, the reference is ViTOL, at 73.17% on CUB and 70.47% on ImageNet.

TRT's figures are Gt-Known numbers whose thresholds were not necessarily chosen under the leakage-free protocol. We will therefore retrain TRT and ViTOL from their published descriptions on the same backbone, the same pretrained weights, and the same number of epochs. Each method, ours included, gets the same hyperparameter search budget, and thresholds are selected only on the held-out split that Choe et al. prescribe. We report MaxBoxAcc-V2, Gt-Known, and Top-1 Loc for all methods.

Beating TRT means one of two outcomes. Either we match its retrained MaxBoxAcc-V2 within run-to-run noise (three seeds) while using a single pass with measurably lower FLOPs, wall-clock latency, and peak memory at batch sizes 1 and 64, or we exceed it at equal cost.

The ablations isolate the key ingredient. Semantic-only admission ($\alpha \equiv 0$) is essentially single-pass TRT-style pruning. Contrast-only admission ($\alpha \equiv 1$) tests the opposite extreme. A fixed blend ($\alpha = 0.5$) tests whether handover matters or plain fusion suffices. A global epoch-based schedule tests whether per-image pacing matters. Random admission at the same $K$ separates the effect of having fewer tokens from the effect of choosing them well. Finally, an SGAN-style variant runs full attention and multiplies contrast into the attention map afterward. It tests whether contrast helps only as a mask, with no pruning involved.

Why not just…?

Why not just prune with the network's own class attention and skip contrast entirely? This is the most serious threat. It is simpler, it is what a single-pass TRT would look like, and on CUB, where birds dominate the frame, it may work well. Our hypothesis says it should fail in one specific way: lower coverage of non-discriminative parts early in training, which becomes permanent because pruned patches never reach the localization map. The checkpoint coverage measurements and the $\alpha \equiv 0$ ablation decide this directly. If semantic-only admission matches the full model on both datasets, the contrast component is not earning its place. The honest result would then be "single-pass semantic pruning is enough," and we would report it that way. We expect any difference to be larger on ImageNet than on CUB, because ImageNet has more off-center and small objects.

Why not just use a center Gaussian, or off-the-shelf saliency boxes as pseudo-labels? Choe et al. show the center prior is embarrassingly competitive on CUB, so it is a real baseline rather than a straw man. It cannot follow off-center or multiple objects, though, and ImageNet results will show how much that matters. Raw saliency pseudo-labels inherit saliency's systematic errors. The SPCL ablations found saliency-derived proposals alone underperformed the curriculum version, which reached 29.96% AP on PASCAL VOC 2007, and SGAN documents clutter and color-match failures. Our design uses contrast only to decide which patches get compute, and only while the network is unreliable. The disagreement-split analysis shows whether that difference matters.

Why not just annotate 5–10 boxes per class? Choe et al. report 92.0% for this few-shot regime on CUB against 70.8% for the best weakly supervised method of the time. That is a real argument against the whole field, and we do not dismiss it. TRT's 94.1% Gt-Known suggests the gap has narrowed, but the two figures come from different protocols. We will train a few-shot box regressor on the same backbone under the same protocol, so that our numbers are read against it and not only against other weakly supervised methods.

Why not just replace attention with fixed local Gaussians for the savings? You et al. found that hard-coded attention loses accuracy on long-range relations, with steep drops as dependency distance grows and 5–10 BLEU losses when cross-attention is hard-coded. Localizing a whole object requires exactly that kind of long-range binding between distant parts. We therefore keep learned global attention among admitted tokens. A fixed-Gaussian variant of the deep layers is cheap to add as a further ablation if reviewers want it.

Decisive Checks

Three outcomes would settle the proposal.

The first is the coverage curve. At early checkpoints, contrast admission must keep substantially more ground-truth box area than semantic admission at equal $K$. Without that, there is no mechanism.

The second is the main comparison under the leakage-free protocol. At a budget near $N/3$, the full model must match or beat retrained TRT on MaxBoxAcc-V2 on both CUB-200-2011 and ImageNet-1K. It must also run in one pass with lower measured latency and memory, and the semantic-only and fixed-blend ablations must fall measurably short.

The third is the disagreement split. On test images where contrast ranks background above the object, the per-image handover must recover performance relative to contrast-only admission. If the second check holds but the third fails, contrast is helping only on easy, well-framed images, and the claimed benefit of the curriculum is not supported.