Place-recognition systems match a new photo to a database of photos with known locations. They are trained almost entirely on carefully curated street-level collections, and the internet holds billions of photos whose location labels are cheap but often wrong. Those labels come from GPS tags, from city names in captions, and from user annotations. The standard way to clean such labels borrows from web-scale data filtering: keep a sample if its nearest neighbors in some embedding space agree with its label. This note argues that the method has a blind spot, and the blind spot hurts place recognition most. When the neighbors are found by appearance, a correctly labeled night or snow photo of a street sits among other night or snow photos from elsewhere, not among daytime photos of the same street. It gets flagged as noise and deleted. The filter therefore removes the hard-condition examples a robust place-recognition model most needs.
The proposal is to build the neighbor graph from scene structure instead of appearance, which should fix this. Structure here means building outlines, line layouts, depth and surface orientation, signals that stay stable when lighting, weather and season change. The rest of the cleaning procedure stays the same. If an image's structural neighbors share its location label, keep it; if they point elsewhere, flag it. The bet is that this catches wrong geotags as reliably as appearance consensus while keeping far more of the correctly labeled night, winter and unusual-viewpoint images. Downstream place-recognition models trained on the cleaned data should then degrade less across conditions.
Core Claim
Neighborhood-consensus label cleaning is only as good as its neighbor graph. For location labels, an appearance-based graph splits apart by imaging condition, so its false flags fall unevenly on rare conditions. A structure-based graph links images of the same place across conditions. It should therefore cut condition-correlated false flags without losing recall on true geotag errors. The measurable claim has two parts. First, at matched precision for detecting corrupted labels, structural consensus retains significantly more correctly labeled hard-condition images than appearance consensus. Second, this retention gives better cross-condition recall for a place-recognition model trained on the filtered pool.
The two directions in the file play different roles. Web-scale noise handling supplies the method and the failure mode. Geometric feature robustness supplies the fix. LEMoN detects mislabeled image-text pairs by combining direct cross-modal distance with k-nearest-neighbor agreement. It beat 12 baselines by more than 3% AUROC on classification noise and by up to 6.3% on captioning noise, and it is the procedure we adapt. Who's in and who's out? motivates the worry: it shows that appearance-driven CLIP filtering of DataComp purges Global South and non-stock imagery at disproportionate rates. We expect the same mechanism to purge non-canonical imaging conditions. The geometric side tells us which signals can serve as condition-invariant neighbors. On the role of geometry in geo-localization shows that textureless building outlines alone support roughly 88% novel-view localization on a 10m grid. UniPR-3D adds 3D layout tokens from a geometry-grounded transformer to DINOv2 appearance tokens and gains about 7% Recall@1 on NordLand summer-versus-winter over a DINOv2 baseline. That gap is direct evidence that appearance features drift apart across seasons.
Research Program
The method has three components. First, a structural descriptor computed from a single uncalibrated image. We start with layout features from a VGGT-style geometry transformer, as used inside UniPR-3D. We pool these features without UniPR-3D's multi-frame aggregation, because web photos are isolated frames. As a cheaper alternative we compute an edge-and-line skeleton in the spirit of the lean images of the geo-localization study, encoded by a small network trained on rendered skeletons.
Second, a consensus score. For image $i$ with location label $y_i$, let $N_k(i)$ be its $k$ nearest structural neighbors. The score is
$$s(i) = \frac{1}{k}\sum_{j \in N_k(i)} \mathbb{1}\left[\lVert y_i - y_j \rVert < r\right],$$
where $r$ is a distance tolerance in meters, set so that images within $r$ of each other plausibly show the same place. Images with low $s(i)$ are flagged. Following LEMoN, we also test a combined score that adds appearance neighbors. When available, we also add text-to-image distance for labels derived from captions.
Third, a verification step for borderline cases. For images near the threshold, we run a two-view structural check against the top neighbors: line-intersection agreement or depth-layout alignment. This adds the certificate-like evidence that the geometric localization literature values, such as the inlier-maximization view of GOPAC. We apply it only to a small fraction of the pool so the cost stays bounded.
We study two noise regimes. In the controlled regime, we take MSLS, which has 1.6M images across 30 cities with day/night and seasonal metadata (Mapillary Street-Level Sequences). We strip its sequence structure so each image stands alone. We then inject realistic corruption at 10–40% rates: geotags swapped to other cities, gross GPS jumps, and distractor images of unrelated content. The ground truth tells us which flags are false, and the per-image condition labels let us measure where those false flags fall. In the real-web regime, we draw images from DataComp CommonPool (DataComp) whose alt-text names a city or landmark, and treat the named place as a noisy location label. There is no ground truth here, so we judge success by downstream place-recognition accuracy, by human audits of samples, and by how much geographic and condition coverage survives filtering. That coverage measurement uses the audit method of the CommonPool exclusion study.
One limitation is built in. Line and edge structure largely disappears in forests and open terrain, as both LDL and FGPL acknowledge by design, and as WildCross documents for depth estimation in natural scenes. We therefore predict the advantage will shrink or vanish there and test this rather than assume otherwise. Structural consensus also cannot correct small GPS drift within $r$. That kind of error is better handled by soft targets, so the proposal targets gross label errors only.
Baselines
The naive reference is no filtering. We train the same place-recognition model on the full corrupted pool. We use a DINOv2 backbone with GeM pooling and the same training schedule as all other runs. Any cleaning method must beat this on cross-condition recall to justify itself. The opposite reference is training on the uncorrupted labels, which bounds what cleaning can recover.
The strongest existing competitor is LEMoN-style consensus using appearance neighbors from DINOv2 or CLIP embeddings. It is the closest published method for label-error detection without retraining, and it is exactly what a practitioner would apply today. We run it with the same $k$, the same $r$, and the same tuning budget for its threshold, all selected on a held-out corrupted validation split. The only difference from our method is the neighbor graph. For this comparison, "beating it" means two things. At equal detection precision, our method must have a lower false-flag rate on correctly labeled night and winter images. And the resulting model must show higher Recall@1 on NordLand and on MSLS night queries, with no loss on overall detection AUROC. LEMoN's paper reports gains over 12 baselines of more than 3% AUROC on classification noise and up to 6.3% on captioning noise. We found no reported result on Lacuna for LEMoN or any consensus filter applied to geotag noise.
For the real-web regime we add a CLIPScore filter between each image and its place-name text, which is the standard DataComp practice. We also add Sieve (SIEVE), which compares a generated caption against the alt-text. The Sieve authors themselves note weakness on geographic and spatial content, which is why we expect it to struggle here; on DataComp-medium it improved average accuracy across 38 tasks by 2.6% over CLIPScore. As a reference point for downstream accuracy, UniPR-3D reports 74.3% Recall@1 on MSLS Challenge and 76.2% on NordLand. Those numbers come from training on curated data, so they serve as a ceiling rather than a like-for-like comparison.
The key ablation swaps graphs while holding everything else fixed: appearance-only, structure-only, and their union. Separately, it toggles the verification step. This isolates whether the gain comes from the structural neighbors themselves or from extra compute spent on verification.
Why not just…?
Why not just use DINOv2 neighbors, which are already fairly robust to conditions? This is the most serious threat. If DINOv2 neighborhoods already bridge day and night, the proposal collapses into a minor gain. The roughly 7% NordLand gap between UniPR-3D and its DINOv2 baseline suggests appearance features still fragment across seasons, but that evidence comes from retrieval, not label cleaning. The per-condition false-flag measurement in the controlled regime settles the question directly. If appearance and structural graphs give similar false-flag rates on night and winter images, we will report that the hypothesis fails.
Why not skip filtering and train through the noise with soft targets? Momentum distillation in ALBEF and label smoothing in Improved baselines for vision-language pre-training both soften one-hot targets so the model memorizes noisy pairs less. We run both on the unfiltered pool under the same budget. We expect them to handle small drift well but not geotags pointing to the wrong city, where the soft target is still centered on a wrong place. They also combine naturally with filtering, so we report that combination as well.
Why not let the place-recognition model filter its own data, keeping low-loss samples? Self-filtering by training loss is a common recipe. We found no reported result on Lacuna for it in place recognition. It has the same structural problem as appearance consensus, because hard-condition images also produce high loss early in training. We include it as a baseline and measure its per-condition false-flag rate alongside the others.
Why not run full structure-from-motion verification on everything? Multi-view reconstruction is how curated sets earn their trust, and in the controlled regime it would likely be the most accurate filter. Its cost scales badly with pool size, and it needs overlapping views that sparse web photos often lack. We run it on a subsample as an accuracy ceiling and report how much of the gap our cheaper scoring with selective verification closes, per GPU-hour.
Decisive Checks
The central experiment is the controlled MSLS corruption study, read along two axes: detection AUROC on corrupted labels, and the false-flag rate on correctly labeled images, broken down by day/night and season. The proposal succeeds only if structural consensus matches appearance consensus on the first axis and clearly improves the second.
The downstream check trains identical place-recognition models on each filtered pool. We evaluate them on MSLS Challenge, NordLand summer-versus-winter, and Oxford RobotCar, focusing on the drop in Recall@1 between conditions rather than the average.
The real-web check filters CommonPool place-name subsets with each method. We compare how much geographic and condition coverage survives, using the audit approach of the exclusion study, and evaluate the resulting models on the same benchmarks.
The limit check repeats the controlled study on WildCross forest imagery. Our prediction is that structural consensus loses its advantage there. If it does, the result defines where the method applies. If it keeps the advantage even there, that would indicate the learned layout features capture more than man-made lines.
References
- LEMoN: Label Error Detection using Multimodal Neighbors
- Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp
- On the role of geometry in geo-localization
- UniPR-3D: Towards Universal Visual Place Recognition with Visual Geometry Grounded Transformer
- Globally-Optimal Inlier Set Maximisation for Simultaneous Camera Pose and Feature Correspondence
- Mapillary Street-Level Sequences: A Dataset for Lifelong Place Recognition
- DataComp: In search of the next generation of multimodal datasets
- LDL: Line Distance Functions for Panoramic Localization
- Fully Geometric Panoramic Localization
- WildCross: A Cross-Modal Large Scale Benchmark for Place Recognition and Metric Depth Estimation in Natural Environments
- SIEVE: Multimodal Dataset Pruning Using Image Captioning Models
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- Improved baselines for vision-language pre-training