A thermal camera sees heat rather than reflected light. Turning a thermal face into a visible photo, so that an ordinary face matcher can compare it against a mugshot or passport gallery, means inventing color and texture the sensor never recorded. Diffusion models, which generate images by gradually denoising random noise, are good at this kind of invention. The trouble is that what they invent spreads. A beard guessed from a warm jaw region can creep onto the cheeks. An eye texture can bleed into the eyebrows. The result drifts away from the person's true identity. The thermal image is poor evidence about texture, but it is often good evidence about where facial parts end. Cold eyeglass frames, insulating hair, and warm nostrils and mouth have distinct heat signatures.
We propose to use that geometry as a wall inside the generator. We first segment the thermal face into regions. We then add penalties to the generator's attention scores so that, during the steps that fix the image layout and only in its middle layers, visual features cannot freely exchange information across region boundaries. The penalty is scaled by how confident the thermal segmentation is. The claim to test is specific: most of the identity loss in diffusion-based thermal-to-visible translation comes from invented detail leaking across facial regions, and blocking that leakage at attention level improves face verification more than feeding the same segmentation in as an ordinary input.
Core Claim
Spatial attention masking has been developed for text-to-image layout control. The masks there are user-supplied and treated as ground truth. Freestyle Layout-to-Image Synthesis sets out-of-mask cross-attention scores to minus infinity. Localized Text-to-Image Generation for Free via Cross Attention Control and Dense Text-to-Image Generation with Attention Modulation modulate attention at inference time; DenseDiffusion only intervenes early in denoising, when layout forms. Seg2Any found that blocking self-attention across regions only in middle transformer layers removes attribute bleeding while keeping global lighting coherent. These papers supply our mechanism.
Thermal-to-visible translation, by contrast, has used global conditioning. T2V-DDPM feeds the thermal image in as a condition and uses a masked coarse image only to initialize sampling. LaPIG conditions a latent diffusion model on whole-image latents. DiVE separates identity from modality in prompt and adapter weights, with no spatial constraint. DiffX is closest in spirit, since it generates RGB–thermal pairs from layouts. However, it injects layout through a gated cross-attention embedder, leaves the denoiser's internal attention unconstrained, and solves joint generation rather than directional translation.
What is new here is not "masking plus thermal." The new element is a mask that is derived from the degraded source modality, so it is itself unreliable. That changes the design in two ways. The barrier must be soft where the thermal segmentation is uncertain, for example where daytime heat washes out contrast. And the right evaluation is identity preservation and measured leakage, not how well the output adheres to the layout.
Research Program
The generator is a pretrained latent diffusion model (Stable Diffusion-class) fine-tuned for thermal-to-visible translation. The thermal latent is concatenated to the noisy latent, which is the standard recipe; this fine-tuned model without barriers is our main control. Thermal-face segmentation comes cheaply from paired, aligned data. On VIS-TH and the ARL polarimetric set, we run an off-the-shelf visible face parser on the visible image, transfer its labels to the aligned thermal image, and train a thermal parser that outputs region labels $r_i$ and a per-pixel confidence $c_i \in [0,1]$ at latent resolution.
Inside the chosen self-attention layers, the pre-softmax score between latent positions $i$ and $j$ becomes
$$s_{ij} = \frac{q_i k_j^\top}{\sqrt{d}} - \gamma\, c_i c_j\, \mathbb{1}[r_i \neq r_j].$$
Here $q_i, k_j$ are the query and key vectors, $d$ is their dimension, $\gamma \ge 0$ is the barrier strength, and $\mathbb{1}[\cdot]$ is one when the two positions lie in different parsed regions. As $\gamma \to \infty$ with $c \equiv 1$, this recovers FreestyleNet-style hard masking. Low-confidence positions stay open, which is how the method tolerates thermal contrast collapse. The same form is applied to cross-attention toward optional region tokens, such as attributes predicted from thermal in the manner of AP-GAN.
Following Seg2Any and DenseDiffusion, barriers act only in middle layers and early timesteps. We sweep both the layer band and the timestep window, because the known cost of aggressive masking is a pasted-together look, and on faces that would mean broken skin-tone continuity. The mask is downsampled to latent grids of 64×64 or coarser. Following MADM's concern with boundary loss at the latent bottleneck (MADM), we dilate boundaries by one latent cell and treat that band as low-confidence rather than letting aliasing cut through thin structures like eyeglass frames.
The central analysis is a leakage measurement, not just output scores. We perturb the thermal input inside one region, for example removing the eyeglass heat signature or warming the jaw. We then measure how much the generated visible image changes outside that region, using an identity-embedding distance and a pixel distance. An unconstrained model should show large out-of-region changes. If barriers work, out-of-region change should drop while in-region change is preserved. This directly tests the premise, which we suspect but have not shown, that bleeding is a real mechanism here.
Baselines
The naive reference point is matching thermal probes directly against visible galleries with a pretrained face matcher, with no translation. SG-GAN reports 69.18% AUC for direct matching on PCSO. We run the same matcher on our splits so every method is scored identically.
The most important control is the same fine-tuned latent diffusion model with no barriers. It uses identical data, steps, sampler, and tuning budget, so that any gain is attributable to the barrier and not to using a large pretrained prior. VTF-Diff's poor showing in When Visible-to-Thermal Facial GAN Beats Conditional Diffusion may partly reflect lacking such a prior, so this control matters.
The key ablation feeds the identical thermal parsing map as extra input channels to the same model, ControlNet-style, with matched parameter count and training time. This isolates "where the segmentation enters" from "having a segmentation at all."
Among existing diffusion methods, T2V-DDPM reports Rank-1 70.5%, VR@FAR=0.1% of 23.5%, and VR@FAR=1% of 45.5% on VIS-TH. We use its protocol and additionally reimplement it under our evaluator. The strongest translation competitor overall is SG-GAN, which injects 11-region parsing priors into a GAN generator and reports 93.08% AUC on ARL. It uses the same kind of region prior as we do, so it is the fair test of whether diffusion plus barriers beats a region-aware GAN. We reimplement it on our ARL split, since no code is listed in the file, and also cite its reported number.
Internal ablations compare hard versus confidence-scaled barriers, all layers versus the middle band, and all timesteps versus early timesteps only. Beating the baselines means higher VR@FAR=0.1% than both the no-barrier control and parsing-as-input on VIS-TH and ARL. The gains must be consistent across seeds, not paired with a worse FID, and accompanied by the predicted drop in out-of-region leakage.
Why not just…?
Why not just feed the parsing map in as an input condition? This is the real threat. Input conditioning tells the network where regions are but does not stop later self-attention from mixing their features. DenseDiffusion's motivation is precisely that restricting only the entry point leaves secondary bleeding. Still, a well-trained conditioning branch could learn to respect boundaries on its own. The parsing-as-input ablation, together with the leakage measurement, settles this. If it matches barriers on verification and leakage, we will report that masking is unnecessary for this task.
Why not just use a region-guided GAN like SG-GAN? SG-GAN is strong, with 93.08% AUC on ARL. It needs a stack of balanced losses, and it lacks the pretrained generative prior that should help on datasets as small as VIS-TH's 50 identities. VTF-GAN's win over conditional diffusion shows GANs can win on sharpness. So we compare against SG-GAN directly rather than assuming diffusion is better. For the visible-to-thermal direction, we also compare against VTF-GAN's reported FID of 47.35 on Devcom ARL, to test whether barriers close the blurriness gap it documented.
Why not just decouple identity with prompts and adapters as DiVE does? Prompt- and weight-level separation is global. It can tell the model who, but not keep a guessed beard off the cheeks. DiVE's gains are on re-identification with no spatial containment. We include an identity-embedding-conditioned variant, which could be combined with barriers, to test whether the two are complementary or redundant.
Why not just use hard masks, as in layout-to-image work? Thermal parses are wrong exactly where the thermal gap bites. We test with synthetic mask corruption (boundary shifts, dropped regions) and on low-contrast images. The confidence-scaled barrier should degrade gracefully where hard masking should break.
Decisive Checks
The experiments run on VIS-TH and the ARL polarimetric set, including standard thermal $S_0$, where AP-GAN reports 84.16% AUC. SpeakingFaces is used for cross-dataset transfer. The metrics are VR@FAR=0.1% and 1%, Rank-1, AUC, FID, and the out-of-region leakage score.
The proposal succeeds on three conditions. First, barriers must reduce measured leakage relative to the unmasked control. Second, that reduction must predict verification gains across the layer and timestep sweep. Third, barriers must beat parsing-as-input at matched cost. If leakage is low even without barriers, the premise is wrong, and the diffusion-versus-GAN gap has another cause, such as data scale or pretrained prior. That would itself be a useful negative result.
We also report sampling time. LaPIG's roughly 70 seconds per pair is the reference. We test whether T2V-DDPM's coarse-image initialization combined with early-only barriers allows fewer steps without identity loss.
References
- Freestyle Layout-to-Image Synthesis
- Localized Text-to-Image Generation For Free via Cross Attention Control
- Dense Text-to-Image Generation with Attention Modulation
- Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
- T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models
- LaPIG: Cross-Modal Generation of Paired Thermal and Visible Facial Images
- Diffusion-based Synthetic Data Generation for Visible-Infrared Person Re-Identification
- DiffX: Guide Your Layout to Cross-Modal Generative Modeling
- Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis
- Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation
- Matching Thermal to Visible Face Images Using a Semantic-Guided Generative Adversarial Network
- When Visible-to-Thermal Facial GAN Beats Conditional Diffusion