Letting the Query Decide Which Scale to Trust: Condition-Gated Pyramid Descriptors for Drone-to-Satellite Geo-Localization

A drone that has lost GPS can still work out where it is by matching its camera image against a library of satellite tiles. Fog, snow and darkness make that matching much harder, and so does the drone's altitude. The usual remedy has two parts: a large single-resolution vision transformer (a network that splits an image into a fixed grid of patches and processes them at one scale throughout), plus either heavy weather augmentation or a generative "clean-up" stage applied before matching. This proposal starts from a different observation. Corruption does not damage all spatial scales equally. Fog and rain mostly destroy fine texture. Snow hides mid-scale road and boundary geometry. A high-altitude view packs hundreds of tiny objects into the frame, so fine-scale detail becomes noise while regional layout becomes the useful signal. A hierarchical transformer is a transformer that, like a CNN, produces feature maps at several resolutions (1/4, 1/8, 1/16 and 1/32 of the input). It already separates these scales for free. What is missing is a matching descriptor that decides, for each individual query, how much to trust each scale.

We propose to build that descriptor and to test whether it holds up on corruption types and altitudes it never saw during training. Concretely, the model pools one descriptor from each stage of the pyramid. A small gating network reads cheap statistics of the query's own features and outputs a weight per stage, and the final descriptor is the weighted concatenation of the stage descriptors. The claim is that this per-query reweighting generalizes better than either a flat backbone trained with augmentation or a fixed multi-scale fusion. The reason is that the gate learns a small, physically meaningful rule ("when fine features look unreliable, lean on coarse layout"), whereas augmentation memorizes the appearance of each specific corruption.

Core Claim and What Is New

The descriptor is

$$d(x) = \big[\, w_1(x)\,\hat p_1(x),\; w_2(x)\,\hat p_2(x),\; w_3(x)\,\hat p_3(x),\; w_4(x)\,\hat p_4(x) \,\big],$$

where $x$ is the input image, $\hat p_s(x)$ is the L2-normalized pooled descriptor from pyramid stage $s$, and $w_s(x) \ge 0$ are gate outputs that sum to one. Satellite gallery images go through the same encoder. The gallery gate is trained jointly but is fed only gallery statistics, which keeps it nearly constant. Under cosine similarity, matching therefore becomes a query-dependent weighted sum of per-stage similarities.

The idea sits next to several existing systems, and each falls short in a specific way. InfoGeo learns object-centric, view-invariant landmarks with an information-bottleneck objective on a flat DINOv2 backbone. Its authors report that landmark attention gets diluted above roughly 300 m because too many small objects compete for it. That is exactly the case where a coarse pyramid stage, which has already merged small entities into regional structure, should help. AGEN bolts a convolutional local-pattern network onto DINOv2 because the flat backbone misses fine detail. It then balances the two branches' losses with a fuzzy PID controller during training, but the mix of global and local evidence is fixed once training ends. MCGF does use a Swin encoder, but only to feed a diffusion-based restoration module. Retrieval itself runs on a flat EVA-02 model, and iterative sampling adds latency. None of these systems lets the scale composition of the descriptor change with the query. The closest scale-aware retrieval work, Hierarchical Attention Fusion, fuses three VGG16 stages with learned attention masks to suppress transient ground-level clutter. It runs on street photography, uses no overhead geometry, and fuses spatially within the image rather than gating whole stages by query condition. ChangeFormer shows that comparing two images separately at every pyramid stage pays off strongly on satellite imagery. We adopt that stage-wise view, but for retrieval of unaligned views rather than for change masks on co-registered pairs.

Research Program

The program begins with a diagnostic that can kill the idea cheaply. We take a hierarchical encoder in the style of SegFormer and train it for plain drone-to-satellite retrieval on University-1652 clean images. We use SegFormer's Mix Transformer because it has no fixed positional encoding and accepts arbitrary resolutions, which matters when altitude changes apparent scale. For every pyramid stage we then measure how much same-location similarity drops between a clean drone image and its corrupted version. We do this for each weather type in University160k-WX and for each altitude in SUES-200. The result is a sensitivity profile: one curve per condition across the four stages. The hypothesis needs these curves to differ by condition. For example, fog should hurt stage 1 most and snow should hurt stages 2 and 3. If every condition degrades all stages in the same proportion, a per-query gate has nothing to exploit, and we would stop and report that negative result. As a check that the pattern is not an artifact of simulated weather, we repeat the profile on Nordland. Nordland's seasonal change is real, although it is ground-level imagery.

If the profiles differ, the second step trains the gated model. The gate is a two-layer perceptron. Its input is per-stage summary statistics of the query: token-norm mean and variance, attention entropy, and, optionally, altitude when flight telemetry provides it. The training objective is the standard retrieval loss on $d(x)$, plus a per-stage consistency term on pairs that show the same location under different conditions. This term pulls $\hat p_s$ of the clean and corrupted views together only at stages the gate currently trusts. The design echoes the geometry-versus-condition split in MDFL, but it uses stage selection instead of an adversarial mutual-information separator. The deployed model is one encoder pass followed by a tiny gate, with no restoration network.

The third step tests the property that would make the method worth using: whether the gate generalizes to conditions held out of training. We train on University1652-plus augmentations with one corruption family removed and evaluate on that family. Snow is the most important held-out case because SUES-200 identifies it as the hardest condition. We also train on low altitudes and test on high ones.

Baselines

The naive reference point is the same hierarchical encoder with only its final-stage descriptor and no gate, trained with the same weather augmentation. It tells us whether a pyramid backbone alone changes anything. The external anchor at this level is the SUES-200 ViT baseline, which reports drone-to-satellite Recall@1 of 59.32% at 150 m and 77.17% at 300 m (SUES-200).

The strongest existing competitor on the weather question is AGEN. It reports 91.71% Recall@1 on University160k-WX, 95.43% Recall@1 for drone localization on University-1652, and up to 97.12% Recall@1 across SUES-200 altitudes (AGEN). It is the strongest competitor because it already combines global and local scales and trains on the same nine-condition augmentation set. We will re-run it from released code where possible, using our splits and the same image resolution. For the held-out-condition experiments it gets the same training budget and the same augmentation set with the same family removed. Beating AGEN means matching it within one Recall@1 point on the standard University160k-WX protocol while exceeding it by a margin larger than seed-to-seed variance (three seeds) on the held-out-weather split, at lower inference cost.

MCGF, with 84.68% Recall@1 and 88.71% AP on University160k-WX (MCGF), is the restoration-based reference. We compare against it on both accuracy and latency. On cross-dataset transfer, InfoGeo reports 95.07% Recall@1 for University-1652 to SUES-200 (InfoGeo). Our altitude analysis will focus on its 300 m failure regime. InfoGeo does not report per-altitude numbers that we can quote, so we will compute them ourselves if code is available.

Two ablations isolate the key ingredient, which is per-query gating. The first concatenates all four stage descriptors with equal weights. The second learns one fixed weight per stage that is shared across all queries. Both use identical encoders, losses and training budgets. An oracle variant chooses weights by the true condition label and gives an upper bound on what the gate could achieve.

Why not just…?

Why not just use a flat DINOv2 with heavy weather augmentation? This is the real threat, because AGEN's 91.71% shows that augmentation already works well on the conditions it covers. Our bet is narrower. Augmentation fits the corruptions it has seen, while the gate encodes a scale-reliability rule that should transfer to unseen ones. The held-out-snow and low-to-high altitude splits decide this directly. If the flat model with augmentation matches us there, the proposal's main claim fails, and we will report it that way.

Why not just restore the image with a diffusion model first? MCGF concedes that iterative reverse sampling is too slow for real-time onboard use. AGEN reports that restoration front-ends can erase or hallucinate fine landmarks. We will measure end-to-end latency on the same GPU. We will also check whether restoration helps on the held-out corruption, since it is equally exposed to conditions outside its training distribution.

Why not just concatenate all pyramid stages, or learn fixed stage weights? If either matches the gated model, the gain comes from multi-scale features alone, and the gate is unnecessary. These are exactly our two ablations. The diagnostic profile predicts the outcome in advance: gating can help only to the extent that different conditions damage different stages.

Why not just enforce invariance on the final layer, the way SeCo does? Seasonal Contrast obtains condition invariance by contrasting real satellite revisits at the final representation, and it improves BigEarthNet mAP to 87.81% from 86.74% with ImageNet pre-training. That shows final-layer invariance is useful. It also means that forcing everything to be invariant at one depth may wash out the fine detail that clean, low-altitude queries need. We include a final-layer-only consistency variant under the same budget to test this.

Decisive Checks

The idea stands or falls on three results. First, the per-stage sensitivity profiles must differ across weather types and altitudes, both on University160k-WX and on Nordland's real seasons. Second, the gated descriptor must beat equal-weight and fixed-weight multi-scale fusion on held-out corruptions and on held-out altitudes. Third, it must stay within one Recall@1 point of AGEN on the standard University160k-WX protocol while being faster than both AGEN and MCGF. As interpretability evidence, the gate's weights on SUES-200 should shift toward coarse stages as altitude rises. That shift would directly address the high-altitude dilution InfoGeo reports, and it would show that the gate has learned something physical rather than a dataset shortcut.