Test-Time Class-Conditioned Histogram Alignment for Unseen Geographic Regions

Segmentation models for aerial and satellite imagery lose much of their accuracy when deployed on a region they never saw during training. A large part of that loss is radiometric rather than semantic. Roofs, fields and water in the new region are the same kinds of object, but sun angle, haze, sensor calibration and soil color shift their pixel values. The cheapest known fix is to reshape the color histogram of training images toward target-region images, and it beats heavy image-to-image translation networks. It has two limits. It needs target imagery at training time. It also matches whole-scene histograms, so a source tile full of buildings matched to a target tile full of water inherits water's colors.

We propose to move the alignment to deployment time and condition it on the model's own predictions. For each new acquisition, a small differentiable color transform is fitted: one monotone tone curve per spectral band, which cannot move edges or change image geometry. The fitting objective pulls the colors of pixels the segmenter currently calls "roof" toward the training-time distribution of roof colors, and does the same for every other class. Because the segmenter's predictions change as the colors change, predictions and colors are refined together in a loop. The training data need not include the target region, and no labels or coordinates are required at deployment.

Core Claim

The claim is that under geographic hold-out, a label-free, per-acquisition radiometric correction can recover most of the cross-region accuracy gap. It has to be fitted by gradient descent on a multi-part distribution objective with three terms: a class-conditioned sliced Wasserstein distance, an identity prior, and a dynamic-range floor. The correction should close more of the gap than global histogram matching, per-tile normalization, or a single-pass pseudo-label correction. The central hypothesis that makes the differentiable, iterative version necessary is this: on a shifted tile, the segmenter's initial predictions are wrong mostly in exactly the classes whose colors have shifted. A single pass that trusts those predictions inherits their errors. Iterating lets partially corrected colors produce better masks, which in turn produce a better correction. If that hypothesis fails, the method reduces to a one-shot procedure, and the experiments are built to show which is true.

What Exists and What It Leaves Open

The method-supplying and motivating work is Randomized Histogram Matching (RHM). RHM augments source training images by matching their histograms to randomly drawn unlabeled target images, discarding matches that sharply reduce image entropy. On Inria it outperformed Affine, Gamma and HSV jitter, Histogram Equalization, Gray World, ColorMapGAN and CycleGAN. The same paper names its own weakness: it relies on randomly paired images happening to have similar content. We keep its central lesson, that simple statistical alignment beats generative translation. We also keep its entropy safeguard. We change where alignment happens (deployment, not training) and what is aligned (class-conditioned distributions, not whole scenes).

HAQAGen supplies the other half of the method. It shows a differentiable histogram-matching loss working inside a multi-component objective for near-infrared-to-RGB colorization. Its target histogram comes from paired ground truth, however, and it is evaluated only on ground-level datasets. We use the differentiable histogram idea but replace paired targets with source-class reference distributions, and we apply it in an unpaired, cross-region setting.

Two papers constrain the design. GeoMultiTaskNet handles region shift on FLAIR by predicting noisy coordinates as an auxiliary task. It requires unlabeled target imagery and georeferencing, and it does no radiometric alignment. Fine-tune Smarter, Not Harder finds that geospatial foundation models tuned with low-rank adapters generalize well to geographic hold-outs. It also finds that coordinates add negligible gains there. Together these suggest that metadata is not the lever. A strong pretrained backbone is the thing any correction must improve on.

The motivation for region-held-out evaluation comes from two further papers. Rolf et al. report drops of up to 30% on unseen regions. Fraccaro et al. show that random splits overstate deployed reliability.

Research Program

Training is standard. A segmenter $f$ is trained on the source regions. From its training data we store, for each class $c$, a compact reference distribution $P_c$ of spectral vectors. This is kept as sorted projections along a fixed set of random directions, which is all that sliced Wasserstein needs.

At deployment we fit a transform $T_\theta$ to all tiles from one acquisition. This could be one flight strip or one satellite scene, since radiometric shifts are mostly shared within an acquisition. Fitting per acquisition also averages out pseudo-label noise and amortizes cost. $T_\theta$ is a per-band monotone piecewise-linear curve. An optional variant adds a near-identity mixing across bands. Non-spectral channels such as FLAIR's elevation band pass through unchanged.

Let $p_c(x)$ be the segmenter's softmax probability for class $c$ at pixel $x$ after the transform. Let $Q_c(\theta)$ be the confidence-weighted distribution of transformed pixel vectors under those probabilities. Let $\theta_{id}$ be the identity curve and $H$ the per-band entropy of the transformed image. We minimize

$$ \mathcal{L}(\theta) = \sum_c w_c \, \mathrm{SW}\big(Q_c(\theta), P_c\big) + \lambda \lVert \theta - \theta_{id} \rVert^2 + \mu \, \max\big(0, H_0 - H(T_\theta(x))\big), $$

where $\mathrm{SW}$ is sliced Wasserstein distance and $w_c$ down-weights classes with few confident pixels. The identity prior keeps the correction small. The entropy floor $H_0$ is the differentiable analogue of RHM's resampling rule and stops the curve from flattening shadows or clouds.

Gradients flow both through the pixel values and through $p_c$, which is what couples the masks and the colors. We run a fixed small number of steps, with the budget reported, and the segmenter's weights stay frozen throughout.

Two design properties matter. The monotone per-band curve preserves edges and object geometry by construction, so no boundary loss is needed. Class conditioning makes the objective indifferent to changes in class proportions between regions, which is precisely where RHM breaks.

Baselines

The naive reference point is the source-trained segmenter applied directly to the held-out region, with no radiometric handling beyond per-band standardization computed on the source.

The strongest augmentation competitor is RHM itself, run in two versions. The first is a target-free variant that matches training tiles to other source-region tiles; this is the fair comparison for our setting. The second is the original setting, with unlabeled target images available at training time. That version sees more data than we do and serves as a demanding reference. RHM reports 0.553 average IoU for one-to-one cross-city adaptation on Inria and 0.590 IoU for Inria→DeepGlobe (RHM).

The strongest cross-region method on FLAIR in the file is GeoMultiTaskNet, at 47.22% mIoU with 10 source and 3 target departments (GeoMultiTaskNet). It uses target imagery during training, so matching it without that access would count as a strong result.

The strongest backbone baseline is Prithvi 2.0 300M with LoRA, which reaches 87.57% mIoU on the Sen1Floods11 geographic hold-out (Fine-tune Smarter). We apply our correction on top of that model to test whether radiometric alignment still helps once the backbone is strong. No reported result was found on Lacuna for test-time adaptation methods on these benchmarks, so an entropy-minimization baseline is run without a cited number.

The key ablations isolate each ingredient. The first uses global rather than class-conditioned matching. The second is a single-pass correction: predict once, then fit the curve in closed form by quantile regression, with no iteration or gradient through $p_c$. The third removes the entropy floor. The fourth replaces the monotone curve with an unconstrained per-pixel color network, which tests whether geometry preservation matters.

All methods share the same backbone, source data, and label budget. Hyperparameters, including our $\lambda$, $\mu$ and step count, are tuned on a held-out source region, never on the target. We count it as a win if our method beats target-free RHM and the single-pass ablation in mean IoU across at least three independent region splits, by more than the spread across seeds.

Why not just…?

Why not just use RHM augmentation at training time? RHM is the real incumbent and may already buy most of the robustness available. Its limits are structural, though. With no target data, it can only simulate shifts that exist within the source regions. With target data, it still requires the target region to be known in advance. It is also blind to class composition, a failure its own authors describe. We run target-free and target-aware RHM, and we also stack RHM with our correction. If stacking gives nothing beyond RHM alone, the test-time correction is not worth its cost.

Why not just fit class-conditioned histogram matching once from pseudo-labels? This is the most serious threat, because it is simple, non-differentiable and fast. Our claim that iteration matters rests on pseudo-label errors being concentrated in the shifted classes. We test that claim directly. We measure pseudo-label accuracy per class before and after each iteration. We also report the gap between single-pass and iterative correction as a function of how far the region is radiometrically from the source, measured as source-to-target sliced Wasserstein distance. If single-pass matches iterative everywhere, we will report that the simpler procedure suffices and drop the differentiable loop.

Why not just normalize each tile, or use Histogram Equalization or Gray World? RHM already showed that Histogram Equalization and Gray World underperform histogram matching on Inria and DeepGlobe. Per-tile standardization also removes real information, since a flooded tile really is darker than a dry one. We include per-tile standardization as a baseline anyway, because it costs nothing.

Why not just use a geospatial foundation model with LoRA? On Sen1Floods11 this approach is already strong on geographic hold-outs. It may leave little radiometric error to fix. Applying our correction on top of it answers the question directly. A null result there would narrow the method's value to cases without such backbones, such as very high-resolution aerial RGB-NIR imagery.

Proposed Experiments

We evaluate on three benchmarks, always with held-out regions.

On FLAIR, we train on 10 departments and test on held-out departments, reporting mIoU. We also run a leave-several-out protocol to obtain splits with graded radiometric distance from the source.

On Inria, we train on some cities and test on unseen ones, then evaluate the Inria-trained model on DeepGlobe, reporting building IoU. This is the setting in which RHM is strongest.

On Sen1Floods11, we train on the standard split and test on the geographic hold-out set, reporting mIoU. This tests whether per-band curves transfer to radar backscatter.

Beyond accuracy, we report per-class IoU changes. These should concentrate in classes with shifted spectra, such as bare soil, roofing materials and water, and their absence would argue against the mechanism. We also measure boundary quality near object edges to confirm that the monotone transform preserves geometry. We report compute per acquisition as well.

Three checks decide the question. The first is a positive correlation, across splits, between source-to-target radiometric distance and our gain over the naive model. The second is a clear margin over global matching on region pairs whose class proportions differ strongly. The third is a measurable iterative-over-single-pass gain, accompanied by rising pseudo-label accuracy across iterations. If the second check fails, class conditioning is not the lever. If the third fails, the contribution shrinks to a one-shot test-time procedure, and we will say so.