When a segmentation model trained on one set of images is deployed on images from a different camera, city, or rendering engine, much of the damage comes from color. Sensors respond differently, the atmosphere tints scenes, and shadows darken whole surfaces. One family of fixes learns to recolor source images so they look like target images, then retrains the model on the recolored images using the original labels. This only works if recoloring leaves the scene's layout untouched. Free-form image generators break that rule, and so do the labels. ColorMapGAN enforces it in the most extreme way: a single scale-and-shift per color band, applied to every pixel. That is safe, but the same correction then has to serve sunlit roofs, shadowed streets, and vegetation alike.
We propose a middle ground. Mean-shift is a clustering method that moves each pixel toward the nearest peak of the joint color-and-position density, and it naturally splits an image into regions of coherent color. Our generator uses those regions as its only spatial vocabulary. It learns a separate color transform for each region, sets each transform from that region's color statistics, and blends neighboring transforms only across the soft region boundaries. The output pixel grid stays exactly where it was, so source labels remain valid by construction. Different surfaces, and the same surface in sun versus shadow, can still receive different corrections. A known weakness of mean-shift clustering becomes useful here: it splits shadowed and sunlit road into separate regions (Using Superpixels for Road Segmentation in Street View Images). That split is a problem when regions must carry semantic labels. For color correction it is an asset, because those two regions genuinely need different corrections.
Core Claim
An adversarially trained color transform keeps ColorMapGAN's guarantee that geometry is preserved, but it can fit illumination-dependent domain shift that a global transform cannot. The condition is that the transform is piecewise-affine over mean-shift regions and conditioned only on region statistics. The gain should concentrate where illumination varies within a scene: shadowed and sunlit surfaces, mixed materials, and satellite tiles with haze gradients.
Concretely, let $x_p$ be the Lab color of pixel $p$. Mean-shift on the source image gives $K$ modes and soft memberships $w_{pk}$, which come from the mean-shift kernel in joint position-and-color space and sum to one over $k$. The recolored pixel is
$$x'_p = \sum_{k=1}^{K} w_{pk}\,\big(A_k x_p + b_k\big), \qquad (A_k, b_k) = h_\theta(s_k),$$
where $A_k$ is a 3×3 matrix and $b_k$ a 3-vector forming region $k$'s color transform. Here $s_k$ is a short descriptor of region $k$: its mean and covariance in Lab, its area, and its mean luminance relative to the image. The function $h_\theta$ is a small network shared across all regions. The generator never sees pixel layout, only region summaries. It therefore cannot draw a tree or move a building edge. The only spatial effect it can have is smooth blending, and the blending weights follow color edges the image already contains. A convolutional patch discriminator, as in ColorMapGAN, judges whether recolored source images look like target images. The segmenter is then trained on recolored source images with the original masks.
Relation to Prior Work
ColorMapGAN supplies the method we extend and the safety argument we keep. Its evidence also motivates the work: CycleGAN's free translation collapsed to 14.22% Overall IoU on Béziers→Roanne while the constrained color map reached 42.88%. Translation-based methods supply the cautionary contrast. CyCADA pairs a convolutional generator with cycle and semantic-consistency losses to suppress label flipping. MADAN adds several more discriminators for multiple sources and still fails on thin classes such as poles and fences. Both treat label preservation as something to penalize, whereas we make it structurally impossible to violate. The thin-class failures are a specific prediction to test: a transform that never moves pixels cannot erode a pole.
Output-space adversarial alignment (Learning to Adapt Structured Output Space for Semantic Segmentation) aligns predicted label maps but processes raw target colors. We treat it as complementary rather than competing, and test the two stacked. Domain-Adversarial Training of Neural Networks supplies the general minimax recipe. Superpixel active adaptation (Efficient Active Domain Adaptation for Semantic Segmentation by Selecting Information-Rich Superpixels) shows that color-coherent regions are good units for adaptation, but it uses them to choose what humans should label. We use them, without any labels, as the domain over which a color transform is defined. The color-clustering literature, including the CIELAB skin-pixel work in Beyond Skin Tone and Lab K-means palettes in Palette-Based Color Harmonization via Color Naming, motivates the choice of Lab and the use of region statistics. Neither line uses adversarial training.
Research Program
The first step is to build the region-wise color generator: mean-shift with a fixed bandwidth computed once per source image, the soft membership weights, and the shared network $h_\theta$. Everything except $h_\theta$ is fixed preprocessing, which keeps training close to ColorMapGAN's reported cost of 0.05 seconds per iteration.
The second step varies bandwidth, and with it the number of regions $K$. We will check whether the method approaches global ColorMapGAN as $K\to1$ and becomes unstable or noisy as regions shrink. This curve tells us whether the benefit comes from region structure or merely from extra parameters.
The third step stacks the recolored-source training under a frozen output-space discriminator setup, to see whether input-level and output-level adaptation add up.
A diagnostic study then splits target test pixels by local luminance relative to their surrounding region. This gives a "shadow stratum" defined without new annotation, and per-stratum IoU tells us whether gains appear where the hypothesis says they should.
Baselines
The naive reference is a segmenter trained on source images only. ColorMapGAN reports a U-Net source baseline of 21.68% Overall IoU on Bad Ischl→Villach.
The strongest direct competitor is ColorMapGAN itself. It reports 48.40% Overall IoU on Bad Ischl→Villach and 42.88% on Béziers→Roanne, and it is the method we must beat on its own benchmark. We will rerun it with the same U-Net, the same training schedule, and the same number of hyperparameter trials as our method, rather than relying only on the published numbers. Beating it means a gain on both city pairs that exceeds run-to-run spread over at least three seeds.
On street scenes, the strongest adversarial reference is the multi-level output-space method. It reports 42.4 mIoU on GTA5→Cityscapes with ResNet-101 DeepLab-v2. We will reproduce it with the identical backbone and then add our recolored source on top. Success there means a consistent improvement over the reproduced number, especially on road, sidewalk, building, and thin classes. For context only, MADAN reports 41.4% mIoU in the multi-source setting.
Three ablations isolate the key ingredient. The first replaces mean-shift regions with a regular grid of the same cell count, keeping everything else fixed. This tests whether color-aligned regions matter or whether any local partition would do. The second is a per-pixel color network, a 1×1-convolution generator mapping each Lab value independently. It is spatially safe and nonlinear but blind to region context. The third removes the adversarial loss and fits each region's transform by matching moments to its nearest target palette cluster.
Why not just…?
Why not just use global ColorMapGAN? If a site's domain shift is close to a single global radiometric change, it will match us, and the $K\to1$ curve will show this directly. The case for regions rests on shifts that differ across the scene. Whether Luxcarta's Pléiades pairs contain enough such variation is an open empirical question. For that reason the street-scene experiments and the shadow-stratum diagnostic are essential and not optional.
Why not just use a per-pixel color network? This is the most serious threat. A pixel-wise nonlinear map can already treat dark and bright colors differently, and that may capture much of the shadow correction. It cannot tell a dark asphalt pixel in shadow from an intrinsically dark material in sunlight, because both have the same color. Our region descriptor carries the region's luminance relative to the whole image, which separates those two cases. If the 1×1 baseline matches us in the shadow stratum, the region conditioning adds nothing. In that case we would report this and recommend the simpler model.
Why not just do classical per-region color transfer? Matching each region's statistics to a target palette is cheap and needs no adversary. Like low-level superpixel methods, however, it has no notion of which target colors a given surface should take. A shadowed road might be matched to a dark target building palette. The moment-matching ablation tests exactly this. If it comes within seed noise of the adversarial version, the discriminator is unnecessary.
Why not just skip input adaptation and rely on output-space alignment? That method aligns label layouts but feeds raw target colors to the network. Our stacking experiment measures whether input recoloring adds on top of it. A null result would mean the two address the same error.
Proposed Experiments
The primary experiment runs on Luxcarta Bad Ischl→Villach and Béziers→Roanne with Overall IoU and per-class IoU, compared against our reproduction of ColorMapGAN and the reported 48.40 and 42.88. The secondary experiment runs on GTA5→Cityscapes and SYNTHIA→Cityscapes with mIoU and per-class IoU, comparing source-only, output-space alignment (reported 42.4 mIoU on GTA5→Cityscapes), global ColorMapGAN, the 1×1 color network, the grid partition, and our method. Each is also tested stacked under output-space alignment.
Three checks decide whether the idea holds. Our method must beat the per-pixel color network inside the shadow stratum, not just on average. Mean-shift regions must beat a same-size regular grid. Thin-class IoU, for poles, signs, and fences, must not fall below source-only, which would confirm that the geometry guarantee holds in practice. We expect GTA5→Cityscapes gains to be modest, because that gap involves texture and rendering differences as well as color. A clear win on the satellite pairs and the shadow stratum, together with neutral-to-positive street-scene results, would support the claim.
References
- ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks
- Using Superpixels for Road Segmentation in Street View Images
- CyCADA: Cycle-Consistent Adversarial Domain Adaptation
- Multi-source Domain Adaptation for Semantic Segmentation
- Learning to Adapt Structured Output Space for Semantic Segmentation
- Domain-Adversarial Training of Neural Networks
- Efficient Active Domain Adaptation for Semantic Segmentation by Selecting Information-Rich Superpixels
- Beyond Skin Tone: A Multidimensional Measure of Apparent Skin Color
- Palette-Based Color Harmonization via Color Naming