Overcoming the Thermal to Visible Modality Gap

29 concepts30 papers2 proposals

Computer vision has achieved remarkable success in the visible spectrum, largely driven by massive datasets like ImageNet and MS-COCO. However, standard models are fundamentally "fair-weather fans." When darkness falls, or smoke and fog obscure the environment, visible-light (RGB) cameras fail. Thermal infrared (IR) sensors offer a robust alternative because they operate independently of ambient light, capturing the long-wave infrared radiation emitted by objects rather than the visible light reflected by them.

Yet, deploying deep learning models on thermal imagery introduces a profound challenge: the thermal-to-visible modality gap. If you take a state-of-the-art object detector or face recognition system trained on millions of RGB images and feed it a thermal image, its performance plummets.

To understand why this happens and how researchers are fixing it, we have to look at the underlying physics of image formation and how neural networks learn features.

The Physics of the Discrepancy

Why is the domain shift between RGB and thermal so much more severe than, say, the shift between photos and sketches? The answer lies in what the pixels actually represent.

Visible-light images capture reflectance. The gradients, edges, and textures we see are determined by how surfaces reflect external light sources. A neural network trained on RGB data relies heavily on these fine-grained textures, color distributions, and high-frequency details to identify objects.

Thermal images, by contrast, capture emittance. They map the heat signatures and thermal capacity of materials. As a result, thermal imagery exhibits fundamentally different characteristics:

  • Texture Deficiency: Fine-grained surface textures (like the pattern on a shirt or the color of an eye) disappear entirely, leaving low-contrast, smooth regions.
  • Inconsistent Contrast: The contrast in a thermal image depends on temperature differentials. A human body might appear bright white against a cold background at night, but look dark against a sun-baked asphalt road during the day.
  • Distinct Noise Profiles: Thermal sensors introduce unique sensor noise and edge characteristics that do not align with the clean geometric boundaries found in natural RGB images.

Because of these phenomenological differences, standard data augmentation strategies optimized for RGB fail or even backfire when applied to thermal data. For example, as shown in Partially fake it till you make it: mixing real and fake thermal images for improved object detection, transformations like solarization or color inversion disrupt the specific heat-based intensity gradients that thermal sensors rely on, destroying the underlying physical information.

Similarly, foundation models pre-trained on natural images exhibit a severe domain gap when applied to infrared data. As Zhang et al. observe in IRSAM: Advancing Segment Anything Model for Infrared Small Target Detection, adapting powerful vision models requires explicit architectural modifications to account for the unique edge and noise profiles of infrared radiation.

To bridge this gap, researchers have converged on three main paradigms: generative image synthesis, cross-modal knowledge distillation, and spectrum-invariant feature mining.

Generative Cross-Modal Translation

If a model trained on visible light cannot understand thermal features, why not translate the thermal image into a visible one first? This is the intuition behind generative approaches, which use Generative Adversarial Networks (GANs) to "hallucinate" realistic visible-spectrum details from thermal inputs.

Early work by Zhang et al. in Generative Adversarial Network-based Synthesis of Visible Faces from Polarimetric Thermal Faces demonstrated that GANs could synthesize visible-like faces from thermal captures, allowing standard face recognition galleries to match nighttime probes. However, unconstrained synthesis often discards critical identity information or introduces artifacts.

To make this translation effective for downstream tasks like biometrics, the synthesis must be tightly constrained. Researchers achieve this by incorporating structural and semantic priors:

  1. Attribute Preservation: In Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis, Di et al. introduced AP-GAN, which extracts semantic attributes (such as the presence of eyewear or facial hair) from a visible gallery and uses them to guide the thermal synthesis, ensuring that the generated image retains key distinguishing features.
  2. Semantic Guidance: The authors of Matching Thermal to Visible Face Images Using a Semantic-Guided Generative Adversarial Network developed SG-GAN, which injects facial landmark priors (eyes, nose, mouth) into the generative process. This prevents the network from warping the geometric structure of the face during the modality shift.
  3. Generative Priors: Rather than training a generator from scratch, Mei et al. in Thermal to Visible Image Synthesis under Atmospheric Turbulence leveraged the rich, pre-trained generative priors of StyleGAN2. This allowed the network to simultaneously correct environmental distortions (like atmospheric blur) and perform high-fidelity modality translation.
[Thermal Input] ---> [Semantic/Structural Priors] ---> [Constrained GAN] ---> [Synthesized RGB] ---> [Standard RGB Model]

This translation paradigm extends beyond faces to full video sequences. For instance, Yu developed ROMA in ROMA: Cross-Domain Region Similarity Matching for Unpaired Nighttime Infrared to Daytime Visible Video Translation, utilizing cross-domain region similarity matching to preserve temporal flow and structural integrity when translating nighttime infrared video to daytime visible video without paired training data.

Cross-Modal Knowledge Distillation

Pixel-level translation is computationally expensive and prone to hallucination. An alternative approach is to align the latent feature spaces of the two modalities directly, allowing a thermal model to inherit the semantic wisdom of an RGB model without ever generating a visible image.

This is typically achieved through unsupervised domain adaptation and cross-modal distillation. Because labeled thermal data is scarce, researchers leverage large pools of unlabeled, paired RGB-Thermal data.

In Unsupervised Cross-Modal Distillation for Thermal Infrared Tracking, Sun et al. proposed a Cross-Modal Distillation (CMD) framework. The core idea is simple: pass an RGB image through a powerful, pre-trained visible-light feature extractor, and pass the corresponding thermal image through a student network. By forcing the student's feature representations to mimic the teacher's semantic outputs, the thermal network learns to recognize high-level concepts (like "pedestrian" or "vehicle") despite the lack of explicit color and texture in the input spectrum.

[RGB Image]     ---> [Pre-trained RGB Teacher] ---\
                                                   ---> [Feature Alignment Loss]
[Thermal Image] ---> [Thermal Student Network]  ---/

Similarly, Devaguptapu et al. in Borrow From Anywhere: Pseudo Multi-Modal Object Detection in Thermal Imagery demonstrated that "hallucinating" a pseudo-RGB counterpart from a single thermal image allows a network to effectively bootstrap knowledge from massive visual-spectrum benchmarks, bypassing the fundamental differences in signal characteristics.

Mining Spectrum-Invariant Features

A third strategy focuses on forcing the neural network to be color-blind and texture-agnostic from the ground up, focusing purely on features that remain invariant across both physical sensing technologies.

In tasks like person re-identification (matching a daylight RGB capture to a nighttime infrared capture), the network must ignore modality-specific styles entirely. To force this behavior, Wu et al. introduced deep zero-padding in RGB-Infrared Cross-Modality Person Re-identification, allowing the network to automatically learn domain-specific feature adjustments.

Expanding on this concept in Single-modality self-supervised information mining for cross-modality person re-identification, Wu et al. utilized random single-channel masking and dual-model mutual learning. By deliberately stripping away color information during training, the model is prevented from relying on shortcuts (like clothing color) and is forced to extract spectrum-invariant characteristics, such as body shape, structural proportions, and gait geometry.

Summary

The thermal-to-visible domain gap is a consequence of shifting from light reflection to heat emission. Bridging it requires moving away from standard computer vision assumptions. Whether by using generative networks to hallucinate missing textures, distilling semantic knowledge from RGB teachers into thermal students, or stripping away modality-specific details to uncover spectrum-invariant structures, these techniques allow models to see effectively across the electromagnetic spectrum, turning total darkness into a solvable domain shift.

Key Foundational Work

A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference (2010) — contributed 2 concept elements to this direction.
Infrared face recognition: a comprehensive review of methodologies and databases (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_domain_gap_thermal_to_visibl_c6be98ba,
  title = {cross-modal domain gap (thermal-to-visible domain gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-domain-gap-thermal-to-visible-domain-gap/rdr_c6be98babe3c46fe88a0f039952672f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_domain_gap_thermal_to_visibl_c6be98ba_v1,
  title = {cross-modal domain gap (thermal-to-visible domain gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-domain-gap-thermal-to-visible-domain-gap/rdr_c6be98babe3c46fe88a0f039952672f5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake