When we compare a standard RGB photograph to a thermal image or a hand-drawn sketch, the pixel values share almost nothing in common. A blue shirt in visible light might appear as a bright white heat signature in infrared or a series of hatched lines in a forensic sketch. This "radiometric gap" makes traditional computer vision techniques—which rely on matching colors, textures, or gradient intensities—highly unreliable.
The core insight across recent research is that while the appearance of an object changes across sensors, its underlying structure does not. By shifting the focus from absolute values to the internal relationships between parts, researchers have developed a way to match images across disparate modalities using structural invariants.
The Stability of Internal Correlations
The most fundamental way to bypass sensor differences is to stop looking at pixels and start looking at how pixels relate to their neighbors. Seungryong Kim and colleagues pioneered this with the Dense Adaptive Self-Correlation (DASC) descriptor in DASC: Dense adaptive self-correlation descriptor for multi-modal and multi-spectral correspondence.
Instead of recording that a pixel is "dark," DASC records how that pixel correlates with its surrounding patch. Because the spatial layout of an object remains constant, these internal "self-correlations" act as a stable signal. Whether an eye is rendered in pixels, thermal units, or pencil strokes, its spatial relationship to the nose and brow remains the same. This principle of "second-order" correlations—matching the relationships between pixels rather than the pixels themselves—is also central to the tracking framework proposed by Chaoqun Wang 0012 in Cross-Modal Pattern-Propagation for RGB-T Tracking.
Geometry as a Universal Language
In person re-identification (ReID) and face recognition, the gap between visible (VIS) and near-infrared (NIR) light is particularly wide. To bridge this, researchers have turned to human geometry as an anchor.
Jiaxu Leng argues in Shape-centered Representation Learning for Visible-Infrared Person Re-identification that silhouettes and physical proportions are "modality-invariant." While a thermal camera loses the color of a jacket, it preserves the wearer's shoulder width and height. Similarly, Shuanglin Yan uses body shape as a "universal language" in the ShapeSpeak framework, converting geometric maps into textual descriptions to align different sensors.
Beyond simple shapes, skeletal structures provide even deeper invariance. Yunqi Miao demonstrates that treating pose estimation as a core part of the learning process allows models to focus on anatomical constants that do not shift between spectrums On exploring pose estimation as an auxiliary learning task for Visible-Infrared Person Re-identification.
Graph Reasoning and Topological Consistency
If the goal is to match structures, the natural mathematical tool is the graph. In a graph representation, image patches are nodes, and the edges represent the spatial or functional relationships between them.
Chunlei Peng showed that graphical representations of face patches are remarkably robust across artistic and sensor modalities Graphical Representation for Heterogeneous Face Recognition. This idea has evolved into sophisticated "relational" modules. For instance, MyeongAh Cho uses a Relational Graph Module (RGM) to prioritize facial geometry over volatile textures Relational Deep Feature Learning for Heterogeneous Face Recognition.
This graph-based perspective even extends to satellite imagery. When comparing radar (SAR) to optical images, Yuli Sun treats the problem as Graph Signal Processing Graph Signal Processing for Heterogeneous Change Detection Part I: Vertex Domain Filtering. By filtering signals based on local neighborhood graphs, they can detect changes in the environment based on "topological consistency"—if the connections between points in a landscape change, a real-world change has likely occurred, regardless of how different the sensors look.
Why It Works: Moving Beyond Translation
Earlier attempts at cross-modality matching often tried to "translate" one image into the other (e.g., making an infrared image look like an RGB image). However, translation is prone to artifacts and "hallucinations."
The structural approach is more robust because it identifies what is inherently shared. Whether it is the affinity-based priors used by Luigi Tommaso Luppino for unsupervised change detection Deep Image Translation With an Affinity-Based Change Prior for Unsupervised Multimodal Change Detection or the structural probes used by Ruida Xi to handle missing sensors Miss-ReID: Delivering Robust Multi-Modality Object Re-Identification Despite Missing Modalities, the field is converging on a single principle:
Appearance is a function of the sensor, but structure is a function of the object. By modeling the latter, we can recognize the same world through any lens.