Adversarial Detail Only Where the Low-Resolution Image Is Silent

Thermal cameras are low resolution. A common fix is to super-resolve the thermal image, meaning to predict a sharper version of it, using a sharp color (RGB) photo of the same scene as a guide. This works because object boundaries usually look the same to both sensors. It fails when the color photo has texture that has no thermal counterpart, such as painted road lines, printed text, or patterned fabric. The network then "bleeds" that texture into the thermal output as fake temperature edges. Adding a GAN loss makes outputs sharper, but it makes this failure worse. A GAN loss here means a second network, the discriminator, that learns to tell real high-resolution patches from generated ones; it rewards any sharp, plausible detail, and the RGB guide is the easiest place to copy such detail from.

This proposal starts from one observation. Every fake edge a guided model can invent sits in the part of the image that the low-resolution input cannot see. That part is the null space of the downsampling operator: the set of details that vanish when the image is shrunk. Anything visible at low resolution is already fixed by the measurement. We propose a hybrid loss for guided thermal super-resolution built on this split. The output is forced, by construction, to reproduce the low-resolution thermal image exactly. The adversarial loss is applied only to the invisible detail, through a discriminator that sees the RGB guide and is trained on negative examples of guide texture pasted into thermal images. We also propose a bleeding probe, because the field currently has no direct measurement of this failure and reports only PSNR and SSIM.

Core Claim

Let $y$ be the low-resolution thermal image, $g$ the RGB guide, and $A$ the known downsampling operator (bicubic blur then decimation, as in the CIDIS degradation). Let $A^{+}$ be its pseudo-inverse. Any network prediction $f(y,g)$ can be projected so that it agrees exactly with the measurement:

$$\hat{x} = A^{+}y + (I - A^{+}A)\,f(y,g),$$

The first term is fixed by $y$. The second term, $r = (I - A^{+}A)f$, is the null-space residual, and it is the only part the generator is free to invent.

The training objective has three terms. The first is a reconstruction loss on $\hat{x}$. The second is an adversarial loss whose discriminator $D$ sees only the residual $r$ together with the high-pass of the guide $g$. The third term trains $D$ on extra negatives: real thermal residuals with guide high-pass texture added. Written out:

$$\mathcal{L}_G = \lVert \hat{x} - x \rVert_1 + \lambda\, \mathcal{L}_{adv}\big(D(r, g)\big), \qquad \mathcal{N}_D = \{(r_x + \alpha\, h(g),\, g)\} \cup \{(r_{\hat{x}}, g)\}$$

Here $x$ is the high-resolution thermal ground truth, $r_x$ its null-space residual, $h(g)$ a high-pass of the guide's luminance, $\alpha$ a random strength, and $\mathcal{N}_D$ the set of pairs the discriminator must call fake.

The claim is falsifiable. At matched backbone and compute, this objective should reach the sharpness and perceptual quality (LPIPS) of an ordinary GAN-hybrid. Its bleeding index should be no worse than pure regression. Its low-resolution consistency error should be zero by construction. Its PSNR should stay within a small margin of the regression model. If the conditioned null-space discriminator yields no bleeding reduction over a plain LSGAN at equal sharpness, the claim is wrong.

Where This Sits

The closest work is LapGSR, a light Laplacian-pyramid guided thermal network trained with MSE plus a least-squares GAN loss. It reports 37.516 dB PSNR and 0.9504 SSIM on ULB17-VT. Its discriminator judges only the thermal output. It is not conditioned on the guide, it is not restricted to undetermined detail, and it is never tested for texture transfer. We keep its hybrid recipe and change where and how the adversarial gradient acts.

pix2pix and Alvarez-Gila et al. supply the method. The first introduced the L1-plus-conditional-PatchGAN loss. The second applied it to RGB-to-hyperspectral reconstruction, dropped batch normalization to protect radiometric precision, and reached GFC above 0.99. Neither involves a low-resolution measurement that could anchor the output, and neither considers detail copied from the condition as a failure mode.

Three papers motivate the proposal. DiffFuSR names "guidance bias" as an open risk. It also keeps generation out of the non-RGB bands and uses deterministic pyramid fusion there instead. The Sentinel-2 radiometric-consistency study found that a generative single-image model lowered building IoU from bicubic's 0.60 to 0.56, while a radiometric consistency loss helped. That is evidence that unanchored detail hurts downstream use. Our projection turns a soft consistency loss of that kind into a hard constraint.

ThermalNeRF constrains the design. It aligns gradients across modalities but deliberately decouples densities, because materials diverge across the two sensors. Any guide-coupling prior must therefore allow guide edges to be absent in thermal. A learned discriminator can encode that permission, whereas a fixed gradient-matching penalty forbids it.

Ruffino et al. report that hybrid GANs collapse when the reconstruction weight is too small. With exact projection, the low-frequency content cannot drift. We therefore expect a wider stable range of $\lambda$, and we test this directly.

Research Program

We build on a SwinFuSR-style backbone (SwinFuSR): Swin layers, cross-attention fusion of guide and thermal features, and a bicubic residual skip, about 3.3M parameters. We choose it because its results are reported and it is cheap to train. The projection layer is implemented exactly for the bicubic operator, using FFT-domain inversion of the blur on the decimated grid with padding handled by reflection. We check numerically that $A\hat{x}=y$ holds to float precision.

The discriminator is a PatchGAN without batch normalization, following Alvarez-Gila et al. Its input is the residual stacked with the guide's luminance high-pass. Bleeding negatives come from two sources. The first is guide high-pass from the paired image pasted into real residuals, because real scenes contain the exact textures that tempt the generator. The second is synthetic overlays: rendered text, stripes, and checker patches added to the RGB guide only, with the thermal ground truth unchanged. These synthetic overlays are also applied to generator inputs in every compared method, including the regression baselines, so that data augmentation is held equal.

The bleeding probe has two parts. The synthetic part takes the CIDIS test set and adds guide-only overlays at known locations. Its score is the normalized correlation between the change in the super-resolved output and the overlay's high-pass. A perfect model scores zero, and a model that pastes the overlay scores one. The real part is a hand-annotated set of regions in the 100 CIDIS test images where visible texture plainly has no thermal edge, such as signs, markings, and printed surfaces. Its score is excess high-frequency energy in the output relative to ground truth inside those regions. Annotation is done blind to model outputs.

Baselines

The naive reference point is bicubic upsampling, which by definition never bleeds. SwinFuSR reports 25.17 dB PSNR and 0.774 SSIM for bicubic on the PBVS 2024 data. Every method must beat bicubic on fidelity, and we report how far each one falls short of bicubic's zero bleeding.

The strongest reproducible competitor is the same backbone trained with pure L1 regression plus the identical guide-only overlay augmentation. This matters because regression wins the fidelity metrics that the challenges rank on. SwinFuSR reports 28.96 dB PSNR and 0.878 SSIM on PBVS 2024 with 3.30M parameters. On CIDIS ×8 guided, SwinPaste reports 30.94 dB and 0.9201 SSIM, and the best entry in the PBVS 2025 challenge report, CityU HH, reports 31.68 dB. We quote these large-model numbers as external reference points only, because we cannot match their compute or their ensembling.

The adversarial competitor is a LapGSR-style hybrid: MSE plus an unconditioned LSGAN on the full output. We run it both on our backbone and with the LapGSR architecture itself. We also report LapGSR's published 37.516 dB on ULB17-VT as the reference for that dataset.

The ablations isolate each ingredient. One adds the projection to regression, with no GAN. One uses a guide-conditioned discriminator on the full image. One uses a null-space discriminator without bleeding negatives. The last is the full method.

All variants share the training data, overlay augmentation, iteration count, and a hyperparameter budget of eight runs each. Each GAN variant sweeps $\lambda$ over the same grid, and the regression baseline sweeps its learning rate. We report medians over three seeds, because Ruffino et al. show how unstable these models are. Beating the baselines means two things. Against the LSGAN hybrid, the full method must match or improve LPIPS and edge sharpness on true thermal edges while cutting the bleeding index substantially. Against regression, it must improve LPIPS at no greater bleeding, losing at most a few tenths of a dB in PSNR.

Why not just…?

Why not just use regression and accept the PSNR win? This is the most serious alternative. The PBVS 2025 leaders in the challenge report use regression and lead on PSNR. Regression avoids hallucination by averaging, and the cost is blur: pix2pix showed L1-only outputs fool human raters essentially never, versus 18.9% for the hybrid on aerial maps. Our experiment states the question as a trade-off curve between bleeding and sharpness rather than a single PSNR number. If augmented regression already sits at the same point as our method, the proposal fails, and we will report that.

Why not just enforce downsampling consistency, without a discriminator? Back-projection guarantees the output matches the low-resolution measurement, but by construction it leaves the null space untouched. Bleeding lives entirely in the null space, so projection alone cannot remove it; it can only stop low-frequency radiometric drift. The projection-plus-regression ablation measures exactly this, and we predict it will show near-unchanged bleeding scores.

Why not just train with guide-only texture augmentation? This may be enough to teach a regression model to ignore guide texture, which is why the augmentation is given to every baseline. What augmentation cannot supply is sharpness on the real thermal edges the network is uncertain about. The open question is whether adversarial pressure can be added back without undoing the augmentation's effect. The comparison between the LSGAN hybrid with augmentation and our method answers that directly.

Why not use a diffusion model? DiffFuSR itself avoided generating the physical non-RGB bands, which concedes the same risk we target. Iterative sampling also conflicts with real-time drone and night-vision use. Our null-space split could in principle wrap a diffusion sampler, but that is outside this study's scope.

Proposed Experiments

The primary benchmark is CIDIS: 700/200/100 split, ×8 and ×16 guided tracks. We measure PSNR, SSIM, LPIPS, low-resolution consistency error, edge sharpness restricted to ground-truth thermal edges, and both bleeding scores. Secondary benchmarks are PBVS 2024 and ULB17-VT, where SwinFuSR's and LapGSR's published numbers allow cross-checks. VGTSR2.0 tests the method under parallax and fog. On VGTSR2.0, GDNet reports only relative gains, so no reported absolute result was found on Lacuna for that baseline, and we will retrain it ourselves if code allows.

Three checks decide the outcome. First, the synthetic bleeding probe should show an ordering: LSGAN hybrid worst, regression and our method close, bicubic zero. If our method bleeds as much as the LSGAN hybrid, the null-space conditioning does nothing. Second, at ×16, where SwinPaste reports single-stage training hallucinates, we sweep $\lambda$ over two orders of magnitude. We expect the projected model to stay stable across a range where the unprojected hybrid collapses or degrades. Third, we test guide removal and fog-corrupted guides on VGTSR2.0. A model trained to distrust guide-only texture should degrade more gracefully than SwinFuSR's reported 6.63% PSNR drop under guide absence, measured with the same protocol.