Weakly supervised localization is the problem of learning where an object is from labels that only say what the image contains. It is usually treated as ill-posed. A class label like "duck" cannot tell water from bird, so networks latch onto the most distinctive fragment of the object. Instance-level retrieval, which means finding photos of one specific physical object such as a particular cathedral among millions of look-alikes, is usually treated as a harder cousin of this problem. We argue the opposite. When two photos share an instance label, they contain the same rigid physical object, not merely two members of a category. Standard feature matching with a robust geometric fit (RANSAC, which keeps only the point matches consistent with a single camera transformation) then marks the pixels in each photo that belong to the shared object. Instance labels therefore carry a spatial signal that category labels lack.
The proposal is to mine this signal from image-level labels in the Google Landmarks Dataset v2 and turn it into pseudo bounding boxes. We then train a landmark detector on those boxes and use it for regional indexing in place of the 94,000 hand-drawn boxes that the best regional retrieval system required. The question is whether "the region that repeats rigidly across viewpoints" is at least as good a training target for retrieval as the region a human annotator draws. It could even be better, because it is by construction the region that will later match.
Core Claim
Detect-To-Retrieve showed that detecting the landmark before aggregating features reduces indexing to 3–4 regions per image and improves Revisited Oxford Hard mAP by 9.3% over prior methods. Its detector, however, was trained on Google Landmark Boxes (GLB), a manually curated set of 94k images. We claim that the multi-view geometry already implicit in same-instance image pairs can replace these manual boxes at no annotation cost, and that it also sidesteps the two failure modes that dominate category-level weak localization.
The first failure mode is discriminative part collapse, where activation maps cover only the fragment that suffices for classification, as analysed in LBBA. The second is the breakdown of cross-image consistency in the long tail. I²C aligns features across same-class images in a batch and against class centroids held in memory. That approach needs many images per class, yet about 57% of GLDv2 classes have 10 or fewer images (GLDv2). Geometric agreement needs only two images of an instance, and it is supervised by physics rather than by the classifier. That is why we expect it to escape part collapse, although we test this rather than assume it.
The closest prior work does not use this signal. DELG trains its local-feature attention with image-level labels and runs RANSAC only as post-hoc re-ranking, so the geometry never feeds back into where the model looks. Product1M/CAPTURE avoids manual boxes by pasting clean product cutouts onto backgrounds and training a proposal network on the result. No such cutouts exist for buildings or natural scenes, so we use real multi-view data instead. R-MAC pools over fixed grids of square regions with no notion of where the object is. GeoDistill is the nearest methodological cousin: it shows geometric consistency can supervise weak localization, but it does so for ground-to-aerial camera pose estimation rather than object extent in 2D image search.
Research Program
The first stage builds the pseudo-boxes. For each instance class in GLDv2-clean, we match all image pairs with at least two members, capping pairs per class to bound cost. Matching uses off-the-shelf DELG local features and RANSAC with a homography or fundamental-matrix model. In each image, the verified inlier points give a raw region.
A single pair gives a biased region. Inliers cluster on textured façades and miss plain walls and roofs, which is part collapse in geometric form. We therefore aggregate over the per-class match graph. The box for image $i$ is formed from the union of inliers accumulated across all its verified partners, weighted by inlier count, and then fit robustly to a percentile hull. Different partners see different sides and scales, so the union should grow toward the full co-visible extent. Images with no verified partner receive no box and are excluded from detector training rather than guessed at.
The second stage trains a class-agnostic "landmark" detector on these boxes. We use the same Faster R-CNN recipe Detect-To-Retrieve used on GLB, so the only thing that changes is the source of supervision. We then run one or two rounds of self-training. The detector's own confident boxes on unmatched images are added only if they agree with geometric inliers wherever any inliers exist. This mirrors the expectation-maximization loop of LBBA without LBBA's auxiliary fully annotated COCO data.
The third stage plugs the detector into Detect-To-Retrieve's regional aggregation (R-ASMK, which pools local descriptors from the detected regions into one compact vector). It also provides the region set for regional GeM pooling on a global descriptor.
The analysis has two parts. First, we measure how well pseudo-boxes and detector outputs agree with GLB boxes. We use threshold-independent metrics under the protocol of Evaluating WSOL Methods Right: MaxBoxAcc (box accuracy at the best score threshold) and PxAP (pixel-level average precision). Any threshold or hyperparameter is tuned on a small fixed GLB split shared by every method. Second, and more important, we measure what the boxes do to retrieval. We also characterize where the geometric box and the human box disagree, for example on sky, foreground pedestrians, or unmatched sides of a building, and whether those disagreements help or hurt retrieval.
Baselines
The naive reference points are two global descriptors with no localization. One is a ResNet101 GeM descriptor. Revisiting Oxford and Paris reports 64.7 mAP for it on ROxford Medium and 38.5 on Hard. The other is R-MAC's fixed multi-scale grid, which answers the question "does any learned region help at all." Both are retrained on GLDv2-clean with our backbone so that the comparison isolates localization.
Following Choe et al.'s finding that a centered blob beats many weak localizers, we add a Center Gaussian spatial weighting before pooling. On landmarks, where queries are often off-center, we expect it to lose. That is an empirical claim to be tested, not an assumption.
The strongest existing competitor is Detect-To-Retrieve's detector trained on the full GLB boxes, combined with R-ASMK. It is the only system that localizes landmarks specifically for retrieval, and it is the upper bound we are trying to reach without its annotations. The file gives its gain only relatively, as +9.3% ROxf-Hard mAP over the prior best. It reports no absolute mAP in the summary, and no reported result for standalone detection mAP on GLB was found on Lacuna. We therefore retrain it with the identical detector architecture, schedule, backbone, and aggregation, on GLB instead of our pseudo-boxes. We report both systems on ROxf and RPar at Medium and Hard, with and without the one-million-image distractor set (R1M), and on GLDv2 retrieval, where the dataset paper's baseline is 22.30 mAP@100.
We count as beating the weak-supervision question if we come within 1 mAP of the GLB-trained detector on ROxf-Hard+R1M while using zero boxes. Outperforming it would support the stronger claim that geometric regions are better retrieval targets than human boxes. DELG with RANSAC re-ranking is reported alongside as the strongest non-regional system; the file gives only its relative +7.1% on Hard R1M.
Four ablations isolate the key ingredient. The first replaces geometric inliers with boxes thresholded from DELG's attention map, which is the same image-level labels without geometry, that is, classic class-activation-map-style weak localization. The second uses single-pair hulls without graph aggregation. The third uses raw pseudo-boxes at index time without training a detector. The fourth drops self-training.
Why not just…?
Why not just prompt GroundingDINO and SAM for "landmark" or "building"? IDOW found that foundation models are tuned for category-level semantics, that instance detection needed extra metric learning on top, and that the whole pipeline is bounded by the proposal recall of GroundingDINO. A generic "building" box also cannot tell the target cathedral from the adjacent shop. Still, this is a cheap, strong alternative, so we run it through the identical R-ASMK pipeline with prompt and threshold tuned on the same small GLB split. If it matches our detector, then the geometric mining is not needed for landmarks, and we would report that.
Why not just annotate a few boxes? This is the most serious threat. Choe et al. report that few-shot full supervision reached 92.0% versus 70.8% for the best weak localization method on CUB. We therefore train the same detector on GLB subsets of 100, 1,000, and 10,000 boxes and plot retrieval mAP against annotation count, with our method at zero. We also test few-shot boxes combined with our pseudo-boxes. If a thousand human boxes clearly beat us, the honest conclusion is that geometric pseudo-boxes are only a cheap pre-training signal. The combined curve then becomes the main result.
Why not skip the detector and localize at test time by matching each image against its retrieved neighbors? For database images this can work, and it is exactly the "raw pseudo-boxes at index time" ablation. It is expensive, however, since it requires pairwise RANSAC over the index. It also gives nothing for a query, or for a database image without a verified neighbor. We measure the fraction of ROxf, RPar, and R1M images that obtain any verified partner, and compare retrieval under both regimes to show where amortizing into a detector pays.
Proposed Experiments
The decisive experiment is the matched comparison on ROxf and RPar Hard with R1M, where the backbone, detector, and aggregation are all held fixed and only the box source varies: GLB, our geometric pseudo-boxes, few-shot GLB subsets, the foundation-model prompt, DELG attention, and Center Gaussian.
The second experiment measures box agreement with GLB via MaxBoxAcc and PxAP under Choe's tuning protocol. Its purpose is to test directly whether graph aggregation escapes part collapse, which should show up as higher PxAP than single-pair hulls and than attention thresholding.
The third experiment slices results by instance-class size in GLDv2, from 2–3 images up to more than 50. The prediction is that geometric pseudo-boxes hold up in the long tail where centroid-based consistency is known to be unstable. We will implement I²C-style global consistency on GLDv2 to confirm that instability rather than cite it.
The last check is qualitative and quantitative disagreement analysis. On images where our box and the GLB box differ by more than 0.5 IoU, we ask which box yields higher retrieval average precision for queries of that landmark. This check decides whether the region that repeats across views is a better retrieval target than the region humans draw, or merely a cheaper one.
References
- Detect-To-Retrieve: Efficient Regional Aggregation for Image Search
- Boosting Weakly Supervised Object Detection via Learning Bounding Box Adjusters
- Inter-Image Communication for Weakly Supervised Localization
- Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval
- Unifying Deep Local and Global Features for Image Search
- Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining
- Particular object retrieval with integral max-pooling of CNN activations Giorgos Tolias
- GeoDistill: Geometry-Guided Self-Distillation for Weakly Supervised Cross-View Localization
- Evaluating Weakly Supervised Object Localization Methods Right
- Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking
- Solving Instance Detection from an Open-World Perspective