Erasure Sensitivity: Using a Localizer's Own Erasing Branch to Detect Ungrounded Web Captions

Models trained on web image–caption pairs learn to point at objects without ever seeing a bounding box. The captions, though, are often about something the camera never captured: "my favorite Sunday afternoon" under a photo of a dog. A standard trick for making these pointing maps cover whole objects instead of one telltale part is adversarial erasing. You blank out the region the model relied on and force a second branch to match the label anyway. On clean labels this pushes attention from a bird's beak to its wings and body. On web captions it should do harm. When the caption describes nothing in the image, the second branch can only satisfy the objective by latching onto background (grass, water, sky). So the mechanism meant to complete objects instead teaches the model that scenery is evidence.

We propose to turn this failure into a filter at almost no added cost. A caption that is truly grounded in the image should become much less matchable once its supporting region is erased. A circumstantial caption, matched through global scene gist, should barely notice. The erasing branch already computes the erased features, so this "erasure sensitivity" comes for free during training. We use it to decide, pair by pair, whether the expensive part-completing loss is applied at all. The result is one shared backbone with lightweight matching heads. It filters noise and expands localization inside a single training loop, without an offline captioning-and-filtering pass over the whole dataset. At deployment it reduces to a standard vision encoder.

Core Claim

We make two linked claims, both falsifiable. First, on web-scale image–text data, ungated complementary erasing in the style of Adversarial Complementary Learning for Weakly Supervised Object Localization degrades localization quality as caption noise grows, because it rewards background co-occurrence on ungrounded pairs. Second, the drop in image–text match probability caused by erasing the caption's own most-attended region is a better in-loop grounding signal than the model's raw match score. Gating erasure by this signal recovers clean-label-style object completion on noisy data and spends less compute than either ungated erasing or offline filtering.

Concretely, let $p(T \mid I)$ be the in-batch softmax probability that image $I$ is matched to its caption $T$. Let $E_T(I)$ be the image's final feature map with the top-activated region for $T$ zeroed out, as in ACoL's feature-level erasing. The erasure sensitivity is

$$\Delta(I,T) = p(T \mid I) - p(T \mid E_T(I)),$$

and the training loss for a pair is

$$\mathcal{L} = \mathcal{L}_{\text{con}}(I,T) + \lambda \, \mathbb{1}[\Delta(I,T) > \tau_b] \, \mathcal{L}_{\text{comp}}(E_T(I),T).$$

Here $\mathcal{L}_{\text{con}}$ is the ordinary contrastive image–text loss and $\mathcal{L}_{\text{comp}}$ is the complementary loss asking a second matching head to still recognize $T$ from the erased features. $\lambda$ weights the two. $\tau_b$ is a per-batch quantile of $\Delta$ rather than a fixed constant. Using a quantile sidesteps the hand-tuned erasing thresholds that make ACoL and Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation fragile. Pairs below the gate skip the complementary head's backward pass entirely.

Positioning Against Prior Work

The motivating evidence comes from the web-noise literature. Generalizing Image Captions for Image-Text Parallel Corpus documents how often human captions describe non-visual stories and ungrounded entities. Improved baselines for vision-language pre-training shows that hard contrastive targets on such data produce overconfident, poorly calibrated models. Neither looks at what this does to spatial maps.

Two families clean such data, and both supply comparison points rather than method. The first family is offline filtering. BLIP trains a captioner and a filter and rewrites the dataset. SIEVE compares synthetic captions to alt-text with a sentence encoder, to avoid CLIPScore's circularity of judging noise with representations trained on noise. Both pay a full inference pass over the corpus before training. Both judge whole-caption agreement, not whether a caption is spatially supported. The second family softens the targets. Align before Fuse (ALBEF) keeps noisy pairs but distills soft targets from a moving-average teacher, and shows qualitative emergent grounding through Grad-CAM. It reports no box-level localization numbers. Its soft targets reduce the penalty on noisy pairs but do nothing about the discriminative-part problem.

The localization side supplies the mechanism and the constraints. ACoL supplies the erase-and-complete branch, but it was tested only on clean labels. BAS and Weakly Supervised Localization Using Background Images attack part collapse with separate generators or a second full network. That second-network design is exactly the overhead we avoid. The background-image approach also breaks if its "background" images contain objects, which web data guarantees. Unsupervised Object Localization using Generative Adversarial Networks finds that discriminators facing more diverse inputs attend to background. That is consistent with our prediction that forcing a matcher to explain ungrounded captions spreads its attention.

The strongest constraint is Evaluating Weakly Supervised Object Localization Methods Right. It shows that under fair tuning, plain class activation mapping (CAM, which reads localization off the classifier's last-layer weights) matches most successors, and that a fixed center Gaussian beats many of them. We adopt its protocol wholesale, and we treat any gain smaller than its tuning variance as no gain.

On the compute side, MGR and Dual Path Learning accept multi-network training costs because the extra parts are discarded at inference. RAPIER reports heavy overhead from three concurrent generators. Our design shares this train-heavy, deploy-light asymmetry. It differs in that the extra head is skipped per sample during training rather than run on everything.

Research Program

The system is a vision transformer image encoder and a small text encoder trained contrastively on web pairs. On top of the image feature map sit two lightweight matching heads, built from 1×1 projections followed by pooling and dot product with the text embedding. Head A produces a caption-conditioned activation map and the standard match score. The top region of that map is zeroed in feature space. Head B rescores the erased features against the same caption, which yields $\Delta$ at the cost of one head forward and no extra backbone pass. Pairs above the batch quantile receive the complementary loss through Head B. The final map is the element-wise maximum of both heads, as in ACoL.

We also test a soft variant that weights $\mathcal{L}_{\text{comp}}$ by a sigmoid of $\Delta$, and a variant where $\mathcal{L}_{\text{con}}$ itself is down-weighted for low-$\Delta$ pairs. The second variant tests whether erasure sensitivity is useful as a general noise weight, not only as an erasing gate.

The analysis has three parts. The first checks whether $\Delta$ detects noise. We inject controlled caption swaps and "circumstantial" replacements (captions drawn from other images' non-visual text) into a clean subset. We then measure detection area under the ROC curve for $\Delta$, CLIPScore from the same in-training model, and SIEVE's caption-distance score. The second part measures how localization quality degrades with injected noise rate for ungated, gated, and no-erasing training. The third part is an accounting of wall-clock time, peak memory, and the fraction of backward passes skipped, with filtering inference charged to the offline methods.

Baselines

The naive reference point is the same contrastive model with no erasing branch, whose maps are read out CAM-style against a text query of the class name. It is also the localization behavior ALBEF demonstrates qualitatively. Alongside it we run the fixed center Gaussian of Choe et al. as a floor that every learned method must clear, since that paper shows it beats many published methods on CUB.

The strongest existing method for this question is ungated ACoL-style erasing combined with offline SIEVE filtering. That pipeline removes noise before training and then completes objects. To keep it fair, it uses the same backbone, the same web corpus and the same number of training steps. It also gets the same 30-trial random hyperparameter search on the small held-out box set that Choe et al. prescribe. Its total GPU-hours include running the captioner and sentence encoder over the corpus.

We cannot transfer a directly comparable reported figure, because no reported result was found on Lacuna for any method trained on web captions and scored by MaxBoxAcc. For orientation only, Choe et al. report 70.8% localization accuracy for the best weakly supervised method on CUB-200-2011 under their standardized protocol, trained on clean labels. Under the older top-1 error metric, ACoL reports 54.08% and the background-image method reports 51.21% on CUB. Beating the strong baseline means two things. Our model must exceed SIEVE-plus-ACoL on MaxBoxAcc and pixel-wise average precision (PxAP) by more than the seed-to-seed spread on CUB and ImageNet at matched total compute, or match it at clearly lower compute. And it must widen that margin as injected noise increases.

A momentum-distillation baseline in the ALBEF style, with erasing added, tests whether soft targets alone already neutralize the noise. Two ablations isolate the key ingredient. The first gates erasure by the model's own match score $p(T \mid I)$ instead of $\Delta$. The second gates at random with the same skip rate. If either ablation matches the full method, the erasure signal is not what matters. The random one also separates compute savings from accuracy effects.

Why not just…?

Why not just filter offline with CLIPScore or SIEVE and then train ordinary erasing? This is the real threat, and it is our main baseline. SIEVE reports a 2.6% gain over CLIPScore on DataComp medium (128M pairs), so caption-level filtering clearly helps downstream accuracy. But it judges whether a caption agrees with a synthetic description, not whether any region supports it. It also requires a full captioning pass over the corpus. If SIEVE-plus-ACoL equals our localization at equal total compute, the accuracy claim fails and only a compute claim survives. The matched-compute comparison and the noise-detection AUC experiment settle this directly.

Why not just rely on soft targets, as ALBEF does? Momentum distillation lowers the penalty for imperfect pairs, but it never tells the erasing branch to stop. A noisy pair still gets a complementary loss whose only solution is background. Our momentum-plus-erasing baseline tests this. If soft targets alone prevent the background drift seen in the ungated noise curve, gating is unnecessary and we would report that.

Why not just annotate a few boxes? Choe et al. report 92.0% on CUB from 5 to 10 boxed images per class, against 70.8% for the best weak method. On a closed category set, few-shot box supervision wins, and we do not claim otherwise. Our claim concerns training localization from open-vocabulary web text, where there is no fixed category list to annotate. We report the few-shot number as an upper reference, not as a competitor we expect to beat.

Why not just use plain CAM? Choe et al. found plain CAM competitive with erasing methods under fair tuning, so any erasing benefit may vanish. Our no-erasing baseline is exactly this. The prediction that ungated erasing falls below it under noise is part of what we test.

Proposed Experiments

Training uses the web corpora ALBEF uses, Conceptual Captions, SBU and CC12M, at a 4M-pair scale first. A controlled-noise variant is built from a cleaner subset with known corruption rates. Localization is evaluated by text-queried maps on CUB-200-2011 and ImageNet using MaxBoxAcc and PxAP under the Choe et al. tuning protocol. We report PASCAL VOC 2007 CorLoc as a multi-object check, since Teh et al.'s Attention Networks for Weakly Supervised Object Localization report 64.59% there with clean labels and proposals. We also check zero-shot retrieval on MSCOCO to confirm that gating does not damage global alignment.

Three checks are decisive. First, if ungated erasing does not degrade relative to no erasing as injected noise rises, the motivating mechanism is wrong and the project reduces to a filtering comparison. Second, if $\Delta$ does not beat the in-training match score at detecting injected noise, and the match-score-gated ablation equals the full method, the core ingredient adds nothing. Third, if the full method does not beat SIEVE-plus-ACoL at matched GPU-hours, the efficiency argument fails. Every result is reported across three seeds with the full tuning budget, so that improvements are not artifacts of threshold choice.