Evidence-Gated Synthesis for Cross-Area Cross-View Localization

A car or robot can locate itself to within about a meter by matching its street-level camera image against an overhead satellite tile. These cross-view localizers degrade badly when moved to a city they were not trained on. The obvious fix is to generate synthetic training pairs that look like the new city. That fix is expensive, because a diffusion model (a generator that turns random noise into an image by denoising it step by step) must run dozens of steps for every full image. It is also risky, because a generator that repaints a whole scene invents rooftops, facades and road markings that no longer match the true geometry. This proposal starts from a simple observation. In fine-grained matching, the decision rests on a small fraction of each image: building corners, roof edges, curbs, lane markings. If that holds, the generation budget should be spent only there.

We propose evidence-gated synthesis. First, a trained cross-view matcher marks the small image regions it actually uses to establish correspondences. A diffusion model then repaints only those patches in the visual style of the target city. Everything else, including the ground-truth camera pose, stays real. Because the source pairs keep their exact pose labels and the geometry is untouched, the synthetic data cannot corrupt metric supervision. Because only a small fraction of pixels is generated, the cost of generation should fall roughly in proportion. The central empirical question is whether this beats the cheap alternatives under an equal compute budget. Those alternatives are photometric augmentation, which costs nothing, and self-distillation on real target-city images, which needs no generator at all.

Core Claim

Our claim is that most of the city-to-city gap in fine-grained cross-view localization lives in the appearance of the matching evidence itself. Roof materials, facade styles and road-marking conventions differ between cities, whereas the background matters little. It follows that restyling only the evidence patches of pose-labeled source pairs should recover most of the benefit of full-image target-style synthesis, at a fraction of the sampling cost and with no geometric hallucination. The claim is falsified under either of two outcomes. If restyling randomly chosen patches of the same area works equally well, evidence selection is not what matters. If color and texture jitter confined to the evidence patches works equally well, the generator is unnecessary.

Where this sits

Three strands of prior work meet here, and the proposal borrows something different from each.

The fine-grained recognition literature motivates the premise that evidence is sparse. Fine-Grained Crowdsourcing for Fine-Grained Recognition showed that human experts separate confusable subordinate classes while looking at under 10% of an object's surface. DeCO supplies the mechanism we adapt. It scores patches with a teacher's attention, spreads selections spatially, and packs them into 2×2 grids, reaching 65.53% on CUB-200-2011 at one image per class. DeCO discards global geometry, which is harmless for classification but fatal for pose estimation. We therefore keep patches in place and change their appearance rather than recomposing them.

The cross-view localization literature supplies both the evidence detector and the target task. FG2 lifts DINOv2 ground features into 3D, pools them along height, and matches them to aerial points with a dual-softmax, solving the pose in closed form. Its explicit, scored correspondences are a natural readout of "where the evidence is." We treat those match scores as DeCO's attention scores are treated. FG2 itself makes no attempt to adapt to new appearance.

The cross-view synthesis literature supplies the negative motivation. SkyDiffusion conditions Stable Diffusion on a curved bird's-eye projection to generate whole satellite tiles. Skyeyes chains Gaussian-splatting reconstruction, diffusion inpainting and cross-frame attention to render ground video. Sat2Density learns volumetric density with heavy training cost. All three evaluate realism (FID, SSIM, PSNR) rather than downstream localization, and all acknowledge hallucinated facades or roofs. That is exactly the failure that makes full-frame synthetic pairs dangerous as metric training data.

Two adaptation papers set the bar we must clear. Adapting Fine-Grained Cross-View Localization to Areas Without Fine Ground Truth self-trains on target-city pairs with noisy GPS, using mode-only pseudo-labels and outlier filtering. GeoDistill self-distills with field-of-view masking. Neither generates data, and both constrain the proposal: any synthesis method must justify its compute against them.

Research Program

The pipeline has three stages, each of which we will build and measure.

Evidence extraction. An FG2 model is trained on the source cities of the VIGOR cross-area split. For each source pair we aggregate its dual-softmax match confidences back onto image pixels in both the ground and aerial views. A greedy spatial-suppression step borrowed from DeCO then selects patches until they cover a fraction $f$ of each image. We will report the empirical distribution of $f$ needed to retain, say, 90% of the matcher's summed confidence. This measurement directly tests whether the "under 10%" premise from object-centric recognition transfers to street scenes. If $f$ turns out to be large, the cost argument collapses and we will say so.

Target-style restyling. We fine-tune an off-the-shelf latent diffusion inpainting model separately on target-city ground images and on target-city aerial tiles. These are unlabeled; no pose is needed. We then inpaint only the selected patches, at low noise strength, so that edges and layout are preserved while materials and markings shift toward the target style. Two consistency variants will be compared. In the first, each view is edited independently. In the second, both views share a style code drawn from the same target neighborhood. This comparison tells us whether consistent restyling across views matters for learning matching.

Training and accounting. The localizer is fine-tuned on a mix of real source pairs and restyled source pairs, both carrying exact source poses. Every run logs the GPU-hours spent on generation, filtering and training. This makes the object of study a curve of error against compute rather than a single point.

Baselines

The naive reference point is zero-shot transfer: FG2 trained on source cities and applied unchanged to target cities. Every other method is scored as a relative error reduction over it, which is also how the adaptation papers report results.

The strongest existing competitor is target-area self-training as in Adapting Fine-Grained Cross-View Localization to Areas Without Fine Ground Truth. That paper reports a 20% reduction in mean and 15% in median localization error on VIGOR over zero-shot transfer with a CCVPE backbone, at roughly 6 hours on a single V100. We will re-run it on the FG2 backbone with the same target images and the same GPU-hour budget as our method. Beating it means a larger relative error reduction at equal compute.

GeoDistill (GeoDistill) reports over 10% improvement on base models and surpasses fully supervised HC-Net on VIGOR with a DINO backbone. It will be run under the same protocol. The file gives no absolute meter-level VIGOR cross-area number for any of these methods; the only absolute figure available is FG2's 0.75 m mean error on KITTI same-area. We will therefore re-measure all baselines ourselves rather than quote them.

Full-frame restyling is the synthesis baseline: the same inpainting model applied to entire images rather than to evidence patches. It is run at matched sample count, and separately at matched GPU-hours. This comparison isolates the claimed cost saving.

Two ablations isolate the key ingredients. Random-patch restyling edits the same area fraction $f$, but chosen uniformly at random rather than by match confidence. Evidence-patch photometric jitter applies color, contrast and texture perturbations to the same evidence patches with no generator. Our method must beat both to support the core claim.

Finally, we test complementarity: evidence-gated synthesis combined with the self-training baseline, which is the realistic deployment configuration.

Why not just…?

Why not just use ordinary photometric augmentation? Color and texture jitter costs almost nothing and is the most dangerous alternative. It may capture illumination differences between cities, but it cannot turn a gravel roof into a tiled one or change the conventions of lane paint. We will not argue this point in the abstract. The evidence-patch jitter ablation settles it directly. If jitter matches the diffusion variant, the generator is not earning its cost, and the contribution reduces to "augment where the matcher looks."

Why not just generate whole synthetic pairs, as SkyDiffusion does? SkyDiffusion and Skyeyes report realism gains but no downstream metric-localization gains. Both describe hallucinated roofs and facades, and a hallucinated structure paired with a real pose is a wrong label. The full-frame baseline, run at both matched sample count and matched compute, measures whether whole-image generation helps localization at all and what it costs per unit of error reduction.

Why not just self-train on real target-city images and skip generation entirely? This is a real threat. Xia et al. already reduce error by 20% using only noisy GPS and about six GPU-hours. Our answer is empirical and two-sided. Self-training inherits the teacher's errors under domain shift; the paper itself notes a remaining gap to an oracle model. Restyled source pairs, by contrast, carry exact labels. But if self-training alone dominates at equal compute, and the combination adds nothing, we will report that generation is not worth it for this problem.

Why not just use DeCO's patch grids directly? Packing patches into grids destroys the spatial layout from which pose is solved. Evidence selection transfers from DeCO; recomposition does not. That is why we edit patches in place.

Proposed Experiments

The main experiment uses VIGOR (VIGOR) cross-area: train on New York and Seattle, adapt to and test on San Francisco and Chicago. We report mean and median meter error. KITTI serves as a limited field-of-view check, where we report longitudinal and lateral error. Orientation is given as known, matching the protocols of the adaptation papers, since yaw sensitivity is a separate known weakness of FG2.

The first decisive check is the curve of error reduction against GPU-hours for all methods. Evidence-gated synthesis should lie strictly below full-frame restyling at every budget. It should also reach at least parity with self-training at matched compute. Evidence that the premise holds would show up as a generation cost roughly proportional to the measured $f$.

The second decisive check is the pair of ablations. The gap between evidence patches and random patches, and the gap between the generator and jitter, must both be significant across three seeds; otherwise the core claim is rejected.

The third check is geometric faithfulness. Before and after restyling, we measure how far FG2's correspondences move on the edited pairs. This verifies that the edits changed appearance but not the structure the pose depends on. Edits that shift correspondences beyond a small pixel threshold will be filtered out, and the fraction removed will be reported as part of the cost.

A secondary analysis asks whether the benefit concentrates on target-city scenes whose evidence patches look least like the source cities. We will measure this similarity with DINOv2 features. A positive result would directly support the claim that the gap lives in the evidence.