Bridging the Modality Gap in Cross Spectral Vision

847 concepts530 papers3 proposals

When we train a computer vision model on standard color (RGB) images, it learns to rely heavily on surface reflectivity—the rich textures, colors, and lighting gradients of the visible spectrum. However, when that same model is forced to operate at night or in adverse weather, it must often switch to alternative sensors like thermal infrared (TIR) or Synthetic Aperture Radar (SAR).

This transition introduces a fundamental challenge known as the modality gap. An RGB camera captures reflected light waves (~400–700 nm), mapping color and fine texture. A thermal sensor captures emitted heat radiation (~8–14 μm), mapping temperature differentials into a single channel. Because these sensors capture entirely different physical properties, their feature distributions are profoundly misaligned. A red shirt and a blue shirt might look identical to a thermal sensor, while a warm face stands out starkly against a cold background regardless of its visual color.

Directly comparing or fusing these features using standard architectures fails because the domain discrepancy easily overwhelms the underlying semantic information. To overcome this, researchers have developed specialized techniques to align, bridge, or disentangle cross-spectral data.

Designing Invariant Metric Spaces

The most direct way to bridge the modality gap is to force the neural network to map disparate sensor inputs into a shared, cohesive embedding space where distance corresponds strictly to identity or semantic content, not sensor type.

In early foundational work, Ancong Wu et al. introduced the SYSU-MM01 benchmark for visible-infrared person re-identification (VI-ReID) and demonstrated that standard feature descriptors fail across spectrums. They proposed a deep zero-padding network architecture to allow the network to automatically allocate domain-specific and domain-shared channels.

To prevent the network from simply clustering images by their sensor type, specialized loss functions are required. Standard contrastive or triplet losses minimize Euclidean distance, but in cross-modality tasks, the absolute magnitude of feature vectors can vary wildly between domains. To address this:

  • Mang Ye et al. introduced a bi-directional dual-constrained top-ranking loss that simultaneously bounds intra-modality variations and cross-modality discrepancies, ensuring that identity features remain compact across both domains.
  • Hao Yi et al. proposed Hypersphere Manifold Embedding (HSME), which projects features onto a unified hypersphere. By normalizing the feature magnitudes, the model is forced to focus purely on the angular relationship between vectors, stripping away scale differences caused by sensor physics.
  • Xia Li expanded on this geometric intuition in Bi-directional Exponential Angular Triplet Loss for RGB-Infrared Person Re-Identification by introducing an exponential angular triplet loss. This loss penalizes cross-modality alignment errors exponentially based on their angular separation, forcing the network to resolve ambiguous matches where color information is missing.
Standard Triplet Loss:
[RGB Feature] <--- Minimize Euclidean Distance ---> [Thermal Feature]
(Vulnerable to massive distribution shifts in feature magnitude)

Angular / Manifold Alignment:
[RGB Feature] \
               \-- Minimize Angle on Hypersphere --> [Shared Identity Space]
[Thermal Feature]/

Stepping Stones and Intermediate Modalities

Instead of forcing a neural network to bridge the massive chasm between multi-channel color and single-channel thermal maps in a single step, another prominent strategy introduces an intermediate "stepping stone" modality. This stabilizes training by breaking the alignment problem into two smaller, more manageable transformations.

One approach creates an abstract, learned middle ground. Diangang Li et al. proposed an auxiliary "X Modality" generated by a lightweight self-supervised network. By translating RGB images into this intermediate representation, the model creates a uniform data structure that shares statistical properties with both visible and infrared domains, yielding significant accuracy gains.

Alternatively, researchers can use deterministic visual simplifications as a bridge. Hu Lu et al. utilized grayscale images as a progressive bridge within a transformer architecture. Because grayscale retains the structural layout of RGB but discards color information, it serves as an ideal halfway point to align with monochromatic thermal data.

Rather than relying on architectural bridges, generative modeling can also synthesize missing data directly. Guan'an Wang demonstrated in Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification that generative adversarial networks can disentangle identity from style to synthesize cross-modality image pairs. This allows the network to learn fine-grained, instance-level similarities by observing the exact same pedestrian in both synthetic RGB and thermal styles.

Disentangling Content from Sensor Style

An image captured by any sensor is a composite of two distinct factors: its semantic content (the geometry and identity of the objects in the scene) and its modality style (the physical medium of capture, such as thermal radiation or light reflection). The modality gap persists because standard encoders entangle these two properties.

To isolate the underlying semantics, Nan Pu et al. developed a dual-Gaussian variational subspace disentanglement framework (DG-VAE). This architecture explicitly splits the latent space into an identity-specific subspace and a modality-specific noise subspace. By modeling the modality characteristics as distinct distributions, the system can systematically strip away sensor noise, leaving a clean, invariant signature for matching in total darkness.

[Input Image] ---> [Encoder] ---> Subspace 1: Identity Features (Shared)
                             ---> Subspace 2: Modality Style (Discarded)

This decomposition principle is equally vital for multi-sensor image fusion. For instance, Pengwei Liang et al. introduced a self-supervised decomposition approach called DeFusion. By training on a generic dataset with a "Common and Unique Decomposition" pretext task, the model learns to separate shared structural layouts from unique sensor-specific details, allowing it to fuse infrared and visible images seamlessly without task-specific fine-tuning.

Geometry as a Universal Language

When color and texture disappear, what remains consistent across different imaging spectrums? The answer lies in geometry: shapes, structural boundaries, and lines do not change based on the wavelength of the sensor.

This geometric invariance is highly effective for cross-modal alignment. In image registration, Eric Pauwels et al. demonstrated that traditional point-based interest features (like corners) fail to repeat across visible and infrared bands due to differing sensor properties. However, by shifting the focus to line-based geometric analysis, they achieved highly stable image registration, proving that structural contours endure even when local textures completely invert.

In classification and re-identification tasks, models can be trained to seek out these fine-grained structural markers. Pingyang Dai et al. introduced MPANet to discover "cross-modality nuances"—subtle, non-color-dependent features such as clothing length, the presence of glasses, or body contours that remain invariant across both visible and thermal spectrums.

The universality of geometry across modalities is so pronounced that it can even be exploited for security vulnerabilities. Xingxing Wei et al. developed a physical adversarial patch designed to blind both visible and thermal object detectors simultaneously. Because a single color pattern cannot translate effectively across light and heat domains, the authors optimized the physical shape and geometry of the patch using heat-insulating aerogel. The attack succeeded by exploiting a universal feature that both sensor types fundamentally rely on: geometric boundaries.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 562 papers
  2. Version 1 · 2026-10-04 · 633 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_modality_gap_cross_modality_discrepancy_90f507ae,
  title = {modality gap (cross-modality discrepancy)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modality-gap-cross-modality-discrepancy/rdr_90f507aee3f74a1b816aadf0a30db19c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_modality_gap_cross_modality_discrepancy_90f507ae_v2,
  title = {modality gap (cross-modality discrepancy)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modality-gap-cross-modality-discrepancy/rdr_90f507aee3f74a1b816aadf0a30db19c/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake