Translating Radar Into Imagery That Unseen Optical Models Can Read

Satellites that see in visible light go blind under clouds and at night. Radar satellites do not, but the thousands of segmentation models the remote sensing community has trained on optical imagery cannot read radar. One common workaround is to translate radar (synthetic aperture radar, SAR: an active microwave sensor that images through clouds but produces grainy "speckle" and geometric distortions) into a synthetic optical image, then run the existing optical model on it. Current translators are trained to make the output look like the real photo, pixel by pixel or perceptually. That is the wrong target for machine readers. In dense overhead scenes, where small buildings sit edge to edge and flood boundaries wind through clutter, a translation can score well on pixel fidelity and still smear exactly the edges a downstream model depends on.

We propose training the translator with a loss that also asks frozen optical models to segment its output and penalizes their boundary errors. The catch, and the real research question, is that optimizing against one fixed model tends to produce images that fool that model rather than images that are genuinely readable. The core ingredient is therefore to sample the "reader" from a pool of diverse frozen optical models during training. Success is then judged on readers and tasks the translator never saw. If this works, one translator makes an entire library of existing optical models usable on radar without retraining any of them.

Core Claim

A SAR-to-optical translator trained with a composite loss has three parts: pixel fidelity, a perceptual term, and a differentiable boundary loss computed through a randomly drawn frozen optical segmenter. It will raise boundary accuracy of held-out optical segmenters, including ones built for a different task, above what fidelity-only translation achieves. A single-reader version of the same loss will not generalize this way. The claim is deliberately about reusing existing optical models. We do not claim that translation beats a radar-native model trained on the same labels. We expect that model to be very hard to beat on its own training task, and we test this directly.

Where This Sits

Two lines of work supply the method. SINCO trains an image compressor with pixel error plus the Dice (region-overlap) loss of a frozen U-Net segmenter applied to the reconstruction, and improves brain-MRI reconstruction (PSNR 35.69 vs 35.18 dB). It uses a single fixed segmenter, stays within one modality, and never asks whether the benefit transfers to other segmenters. We borrow its "loss through a frozen task model" idea and make transfer the central question. Boundary Loss for Remote Sensing Imagery Semantic Segmentation builds a differentiable stand-in for the boundary F1 score (how well predicted and true edges line up within a small distance buffer). It shows that area losses let adjacent buildings merge, and raises ISPRS Potsdam building IoU to 93.85% from 92.56% (IoU loss) and 92.61% (Dice). That work trains a segmenter on optical input. We instead push its loss back through a frozen segmenter into a translator.

Three translation systems motivate the gap. Any2Any maps five sensors into a shared latent space with a diffusion transformer, a model that generates by iteratively denoising. It trains only on latent regression over fixed 256×256 chips. Diffusion-based SAR-to-EO translation (DSE) evaluates flood segmentation on synthetic optical plus SAR, but only after training, not as part of the translator's loss. Multi-conditional SAR-to-EO GANs add map and coordinate conditioning and leave downstream utility as future work. None of the three optimizes for what a machine reader extracts.

Two works constrain the design. On The Classification-Distortion-Perception Tradeoff proves that distortion, realism and task accuracy trade against each other. Its Theorem 5 says that when the degradation is known, no restoration can lower the Bayes classification error below that of the raw input. Taken seriously, this means translation cannot add information beyond the SAR itself. Any benefit must come from making information accessible to a fixed reader that cannot otherwise use radar. That is why we frame the contribution around frozen readers and include the radar-native model as a reference ceiling. GeoCrossBench shows that foundation models evaluated on bands they never saw lose a factor of 2 to 4 in performance. This argues against hoping an optical model simply reads SAR directly.

Research Program

The translator is a conditional diffusion model in the style of DSE, trained on co-registered SAR–optical pairs. Its objective is

$$\mathcal{L} = \mathcal{L}_{\text{pix}}(\hat{x}, x) + \alpha\,\mathcal{L}_{\text{perc}}(\hat{x}, x) + \lambda\,\mathbb{E}_{s \sim \mathcal{S}}\big[\, w\,\mathcal{L}_{BF1}(s(\hat{x}), y_s) + (1-w)\,\mathcal{L}_{\text{IoU}}(s(\hat{x}), y_s)\big].$$

Here $\hat{x}$ is the synthetic optical image and $x$ the real one, and $\mathcal{L}_{\text{pix}}$ and $\mathcal{L}_{\text{perc}}$ are pixel and perceptual (LPIPS) distances. $\mathcal{S}$ is a pool of frozen optical segmenters with differing architectures, training data and tasks. $s$ is one segmenter drawn per step. $y_s$ is the target mask for that segmenter: ground truth where it exists, otherwise $s(x)$, the segmenter's prediction on the real optical image. Finally, $\mathcal{L}_{BF1}$ and $\mathcal{L}_{\text{IoU}}$ are the boundary and overlap losses of the Boundary Loss paper, including its schedule in which $w$ grows over training.

Because diffusion sampling is long, we apply the task term to a one-step estimate of the clean image at randomly chosen noise levels, a standard trick for differentiating through diffusion models. We verify that it correlates with full-sampling outcomes on a validation split.

The reader pool is the key ingredient. It mixes convolutional and transformer segmenters for flood water, buildings and land cover, trained on optical data only and never updated. Readers are split into a training pool and a held-out pool, with a whole task (for example, buildings) held out in some runs. Using $y_s = s(x)$ as the target means the translator is asked to make readers behave on synthetic imagery as they would on the real image. This turns the method into distillation of the optical view's readability, and it needs no labels for unseen tasks.

Scene complexity enters as the evaluation regime rather than as extra machinery. We stratify all results by object size and building density, and report boundary F1 separately for touching-building clusters. This is where fidelity-only translation should fail and where the claim lives or dies. We do not add entropy-gated fusion in the style of the Gated Segmentation Network. That would confound the loss question with an architecture change.

Baselines

The naive reference point is running each frozen optical reader on raw SAR scaled into the optical input range. Given GeoCrossBench's 2 to 4× collapse on non-overlapping bands, we expect it to be near-useless, and it sets the floor.

The strongest existing translation method is a DSE-style Brownian-bridge diffusion translator trained with its published fidelity objective. DSE reports PSNR 27.25 dB and SSIM 0.695 with its denoising module, and flood IoU within 0.0068 of true optical plus SAR. We retrain it on exactly our splits with the same backbone, step count and hyperparameter search budget as our model, so the only difference is the task term. Beating it means higher boundary F1 of held-out readers on Sen1Floods11 and SpaceNet6, with confidence intervals from three seeds that exclude zero, at a PSNR cost we report rather than hide. We also include Any2Any as a second translation baseline if its weights can be run on our splits; the file gives only a relative PSNR gain (over 102% on RGB-to-MS), not an absolute figure.

The radar-native reference is a segmenter trained directly on SAR with the same labels and compute. On Sen1Floods11, the dataset paper reports mIoU 0.3389 for flood water from a SAR model trained on optical-derived weak labels, and 0.4571 for Otsu thresholding on permanent water. For SpaceNet6 buildings, no reported result was found on Lacuna. We expect this reference to match or beat us on its own training task. Our claim concerns the held-out readers and tasks that it cannot serve without being retrained.

Three ablations isolate the ingredients. The first uses a single frozen reader instead of the pool, which tests the fooling hypothesis. The second uses an area-only task term, SINCO's Dice with $w=0$, which isolates the boundary loss. The third applies the boundary F1 surrogate to edges of the real optical image with no reader at all, which tests whether routing through a model matters.

Why not just…?

Why not just train the segmenter on SAR directly, as Theorem 5 suggests? For any single task with labels, this is probably the best option, and our radar-native baseline measures exactly that. It fails the use case we target. Each of the many existing optical models would need a SAR twin trained with its own labels and pipeline. Our held-out-task experiment asks whether one translator serves a new task at zero retraining cost. If the radar-native model trained on flood labels transfers to buildings better than our translator does, the proposal's premise is wrong, and we will report that.

Why not just add an edge loss on the target photo and skip the frozen readers? This is a genuine threat and it is included as an ablation. Image edges and the edges a segmenter uses are not the same: shadows and texture produce strong image edges that readers ignore, while low-contrast building boundaries matter a great deal to them. If the reader-free edge loss matches the pooled-reader loss on held-out readers, the simpler method wins and we would recommend it.

Why not just adapt the optical model to SAR, or use a band-agnostic foundation model? GeoCrossBench shows that even xViT, the best model on its no-overlap protocol, and DINOv3 degrade sharply on unseen bands. Per-model adaptation also reintroduces the per-model retraining cost we want to avoid. For fairness, we fine-tune one representative reader on SAR under the same budget. This shows the price of adaptation per model against one shared translator.

Proposed Experiments

We train on Sen1Floods11 (Sentinel-1 C-band with coincident Sentinel-2 imagery, 446 hand-labeled chips, 10 m) and SpaceNet6 (0.5 m X-band SAR, optical imagery, building footprints). SEN12MS/DFC2020 land cover supplies additional readers. The metrics are boundary F1 and IoU of each reader on synthetic imagery, and the gap between each reader's output on synthetic and on real optical imagery. We also report PSNR, SSIM and LPIPS so the distortion–perception–utility tradeoff is visible. We trace that tradeoff as a curve over $\lambda$, and results are stratified by object size and building density.

The decisive check is held-out generalization. We predict that the pooled-reader translator improves boundary F1 for held-out readers and held-out tasks, while the single-reader version improves its own reader and gives little or nothing to others. The improvement should concentrate in dense, small-object strata. A second check is sensor transfer: a translator trained on Sentinel-1 is evaluated on SpaceNet6 SAR without retraining, which shows whether readability survives a change of radar band and resolution. The third is the Theorem 5 check: on held-out tasks, translation plus a frozen reader should close part of the gap to a radar-native model trained with that task's labels, without ever exceeding it by more than seed noise. Exceeding it would indicate a leak in the evaluation rather than a result.