Is Weakly Supervised Detection Getting Better at Separating Crowded Objects, or Just More Familiar With the Test Set?

Weakly supervised object detectors learn to draw boxes around objects from image-level tags alone, such as "this image contains dogs," with no drawn boxes to learn from. They have a well-known failure called instance grouping error. When several objects of the same class sit close together, the detector draws one large box around all of them. It does this because a loss that only asks "is a dog present?" is fully satisfied by that one box. Recent systems have posted large gains on PASCAL VOC 2007, from 39.3% mean average precision for the original two-stream network to 63.4% once the Segment Anything Model (SAM) and CLIP are plugged in. Over the same period, those components have increasingly been pretrained on web crawls that plausibly contain the very VOC and COCO test images used to score them. Nobody has checked whether the apparent progress on crowded scenes survives on images the components could not have seen.

The obvious check is to evaluate on fresh images. That check is confounded. Fresh crowded scenes may simply be harder, or easier, than the benchmark's crowded scenes, and text-only contamination research has already shown that this kind of difficulty shift can hide or fake a memorization effect. This note proposes a difficulty-matched audit. It measures grouping errors directly on benchmark test images and on newly collected, post-cutoff images. It matches the two sets on the physical variables that make separation hard (how many same-class objects there are, how close together, how occluded, how small), and it calibrates the procedure with deliberately contaminated models. The quantity of interest is the part of a detector's instance-separation ability that disappears when only familiarity is removed.

Core Claim

We claim that grouping error can serve as a sharp, localization-specific probe of benchmark contamination in vision, sharper than whole-image accuracy. Crowded same-class scenes are exactly where a detector without memorized layout knowledge has the least evidence to split instances correctly. Our hypothesis is that detectors built on large pretrained components show a larger merge-rate gap between benchmark and difficulty-matched fresh images than classically pretrained detectors, and that this gap grows with a measurable exposure signal (near-duplicate presence of the test image in a released pretraining corpus). The null result is also informative. If the matched gap is near zero for every model, the field's crowded-scene numbers on VOC and COCO can be trusted, and residual grouping error is a pure difficulty problem. What is new here is the pairing of an instance-level grouping metric with a difficulty-matched, positive-controlled contamination design. Neither the detection literature nor the contamination literature has done this for spatial localization.

What Prior Work Establishes and Leaves Open

The grouping failure itself is well documented, and it motivates the study. Weakly Supervised Deep Detection Networks set up the two-stream design, in which proposal scores are pooled into an image-level prediction, and this pooling creates the incentive to merge. Several later systems explicitly report persistent merging of adjacent same-class instances even after their fixes. These include voting-based refinement in SLV, the teacher-feedback loop in Cyclic-Bootstrap Labeling, and the three-stage pseudo-label pipeline in Salvage of Supervision. Object Discovery via Contrastive Learning shows that picking only the top-scoring proposal can miss up to 60% of target objects on COCO. C-WSL attacks merging with per-class counts, which only work where humans can count at a glance (one to four items). All of these papers measure progress with mean average precision or correct-localization rate, which fold grouping errors in with every other error. None reports a merge rate, and none asks whether test images were seen during pretraining. We supply the missing metric and the missing control rather than a new detector.

Weakly Supervised Open-Vocabulary Object Detection is the system most exposed to our question. It uses SAM masks as category-agnostic proposals and CLIP for matching regions to text, reaching 63.4% mAP and 80.6% CorLoc on VOC 2007. It treats SAM and CLIP as clean feature extractors. We test that assumption specifically on crowded scenes, where SAM's instance masks are doing the separating.

On the contamination side, Both Text and Images Leaked! is the closest work. Its MM-DETECT tool detects multimodal leakage by perturbing answer order or masking caption words, and it finds that older datasets leak more, with COCO captions among those audited. It never touches boxes, counts, or instance separation. Our design differs in two ways: it measures a localization outcome, and it avoids perturbing the test item. Confounders in Instance Variation both constrains and supplies our method. It shows on arithmetic prompts that rephrasing-based detection is confounded by the rephrased item's difficulty (a "masking effect"). It also uses deliberate fine-tuning contamination as a positive control, finding a 71.0% versus 61.4% boost for low- versus high-diversity templates. We carry both lessons into vision. Crowding, gap, and occlusion play the role of template difficulty, and injected test images play the role of the contamination fine-tune. DyVal and the procedurally generated Cognitive Battery motivate controlled generation of test items. We use procedural compositing only to calibrate the difficulty curve, not as the main test set, for reasons given below. Evading Data Contamination Detection constrains us too: perturbation tests are easy to defeat, which is one more reason to rely on exposure measurements and matching rather than on perturbation sensitivity.

Research Program

The first component is a grouping metric. For each ground-truth "cluster" (two or more same-class instances whose normalized gap falls below a threshold), a predicted box counts as a merge if it covers at least a fraction τ of two or more instances in the cluster while matching none of them at IoU 0.5. The merge rate is merges divided by clusters. Its companion, split recall, is the fraction of clustered instances individually recovered at IoU 0.5. Both are reported alongside mAP, and sensitivity to τ and the gap threshold is reported. We will validate the metric on a few hundred hand-judged predictions before relying on it.

The second component is a fresh crowded-scene set. We will collect roughly 3,000 images of the 20 VOC classes, posted after the training cutoff of every pretrained component we audit. Crowded scenes (two or more adjacent same-class instances) will be deliberately oversampled, and every instance will be box-annotated. Each image in both the fresh set and the VOC 2007 test and COCO validation sets gets a difficulty vector: same-class count, gap between nearest same-class neighbours normalized by box size, pairwise occlusion, instance scale, and truncation.

The third component is an exposure signal. Where a model's pretraining corpus is publicly released, we run perceptual-hash and embedding near-duplicate search of every benchmark test image against it. This gives each benchmark image a graded exposure label (exact duplicate, near duplicate, none). For components with unreleased data, only the benchmark-versus-fresh contrast is available, and we will say so.

The fourth component is the matched comparison. We use coarsened exact matching on the difficulty vector, with inverse-propensity reweighting as a robustness check, to compare merge rate and split recall between benchmark and fresh images within each difficulty stratum. The estimated familiarity effect is the matched gap. We then test whether it rises with the exposure label within the benchmark set, holding difficulty fixed. That within-benchmark comparison sidesteps any residual distribution shift between photo sources.

The fifth component is a positive control. We take an open CLIP-style encoder and continue its pretraining on its usual data, either with or without a held-out half of the VOC 2007 test images injected, following the fine-tuning contamination design in Confounders in Instance Variation. We then retrain the same weakly supervised detector on top of each encoder. The audit must flag the injected half and not the held-out half. The size of the injected-half merge-rate drop tells us how large a familiarity effect on grouping can be.

Baselines

The do-nothing reference is the field's current practice: report mAP and CorLoc on VOC 2007 and COCO and read improvements as progress on separation. For the strongest foundation-model system, the reported reference numbers are 63.4% mAP and 80.6% CorLoc on VOC 2007 from Weakly Supervised Open-Vocabulary Object Detection. For the strongest classically pretrained pipeline, they are 64.4 mAP₅₀ on VOC 2007 and 32.8 mAP₅₀ on COCO from Salvage of Supervision. No reported result was found on Lacuna for merge rate or split recall for any weakly supervised detector, so our first deliverable is those numbers on the original test sets. Beating this baseline means showing that mAP rankings and matched merge-rate rankings disagree for at least one pair of methods.

The strongest competing audit is an MM-DETECT-style perturbation test adapted to detection, following Both Text and Images Leaked!. Here, test images are perturbed (mild crops, colour jitter, horizontal flips, moving instances apart by inpainting), and a drop in the fraction of instances that were correct before the perturbation and wrong after it is read as leakage. It is the strongest competitor because it is the published contamination method closest to vision and needs no new data. We run it on the same models and test images with the same compute, and we tune its perturbation strength on the positive-control encoders with the same budget we give our matching thresholds. Our audit beats it if it separates the injected and held-out halves of the positive control with higher AUROC at a matched false-positive rate, and if its effect estimates do not drift when perturbation difficulty drifts.

The key ablation is the unmatched fresh-set comparison: the same fresh images and models, but no difficulty matching. This isolates the proposal's central ingredient. If the unmatched and matched gaps agree, matching was unnecessary and the simpler recipe suffices.

The detectors audited span the pretraining spectrum. The original two-stream network and SLV use ImageNet backbones. Cyclic-Bootstrap Labeling and Salvage of Supervision are strong classical refiners. The open-vocabulary SAM-plus-CLIP detector and C-WSL serve as a count-supervised reference whose merge rate should be low on small clusters regardless of familiarity. Released checkpoints are used where available; otherwise methods are retrained with their published settings on VOC 2007 trainval.

Why not just…?

Why not just search the pretraining corpora for duplicates of the test images and report the overlap? Overlap tells you what was seen, not whether it helped. Also, several relevant corpora are unreleased, a limitation Both Text and Images Leaked! stresses for proprietary multimodal systems. We do run duplicate search, but as the exposure variable inside a difficulty-held-fixed comparison. If exposure turns out to predict merge rate so strongly that matching adds nothing, the within-benchmark exposure analysis will show that, and we will report duplicate search plus stratification as the sufficient tool.

Why not just evaluate on a new, post-cutoff dataset and compare mAP? A fresh set differs from VOC in crowding, scale, and photographic style, so any gap mixes familiarity with difficulty. This is exactly the masking effect documented by Confounders in Instance Variation. This alternative is the unmatched ablation in our Baselines section. If its answers match the matched analysis across all detectors and the positive control, we will say the cheaper protocol is adequate. We expect otherwise, because crowded scenes are exactly where difficulty varies most.

Why not just use procedurally generated scenes, in the spirit of DyVal, where count and gap are set exactly? Composited scenes carry pasting artifacts and destroy the whole-image layout that memorization would act on. They can therefore detect difficulty effects but not familiarity effects. We use them only to trace how merge rate responds to gap and count for each detector, which checks that the metric behaves sensibly and supplies the functional form used in matching.

Why not just add count supervision, as in C-WSL, and stop worrying? Counts reduce merging for small groups but do not address the measurement question: whether reported separation ability is genuine. C-WSL's own reliance on at-a-glance counting also caps it at small clusters. Including C-WSL as an audited model tests whether count-supervised separation is itself familiarity-dependent.

Decisive Checks

The study succeeds as a tool if the positive control is cleanly recovered. The injected half of VOC 2007 test should show a lower merge rate than the held-out half under the contaminated encoder, with no such difference under the clean encoder, and the perturbation baseline should recover this less reliably. The scientific question is settled by the matched gap and its trend with exposure. A positive matched gap that is larger for the SAM-and-CLIP detector than for ImageNet-backbone detectors, and that rises with near-duplicate exposure, would mean part of the recent crowded-scene progress is familiarity, and that VOC and COCO overstate instance separation. A matched gap indistinguishable from zero across detectors, while the positive control is still detected, would clear the benchmarks and redirect effort to the difficulty curve itself. Either outcome yields a reusable fresh crowded-scene test set and the first per-method merge-rate and split-recall numbers for weakly supervised detection.