In computer vision, matching images across different sensors—such as a daytime color camera (Visible) and a nighttime thermal camera (Infrared)—is often described as a "nightmare" task. The distribution gap between these modalities is not just a matter of color; it involves fundamental differences in pixel intensity, texture, and noise.
Traditional approaches try to force the neural network to project both modalities into a single, shared feature space. However, the distance between these domains is often too large for stable optimization. A growing body of research suggests a more effective strategy: instead of jumping across the chasm, we should build a bridge. By creating a synthetic intermediate modality, researchers are providing models with a "stepping stone" that makes cross-modal alignment significantly easier.
The Intuition of the Stepping Stone
The core problem in Visible-Infrared Person Re-Identification (VI-ReID) is that the model must learn features that are "identity-specific" while ignoring "modality-specific" traits. When a model is trained on two highly disparate domains, it often takes the path of least resistance, focusing on the obvious modality differences rather than the subtle person-specific details.
Diangang Li and colleagues introduced the concept of an "X Modality" in Infrared-Visible Cross-Modal Person Re-Identification with an X Modality. The intuition is simple: if the transition from $A$ to $B$ is too steep, introduce $X$ such that the transitions $A \to X$ and $X \to B$ are manageable. This "X" representation acts as a mediator, breaking a large cross-modal transition into two smaller, more manageable transformations.
How to Build a Bridge
Researchers have developed several ways to synthesize these intermediate representations, ranging from simple pixel-level fusion to complex generative models.
- Pixel-wise Fusion and Channel Augmentation: Xi Yang proposed Syncretic Modality Collaborative Learning (SMCL) in Syncretic Modality Collaborative Learning for Visible Infrared Person Re-Identification, which creates a virtual modality by performing pixel-wise fusion of features. Similarly, Mang Ye demonstrated in Channel Augmented Joint Learning for Visible-Infrared Recognition that treating augmented data as a distinct intermediate modality—rather than just noisy versions of the original—helps the model learn a more robust, modality-invariant space.
- The "Broker" Modality: In the context of crowd counting, Haoliang Meng and Xiaopeng Hong introduced a "Broker Modality" in Multi-modal Crowd Counting via a Broker Modality. This broker provides a middle ground in pixel intensity distribution, which facilitates smoother feature extraction and eliminates artifacts like "ghosting" that occur when sensors are misaligned.
- Implicit and Transition Modalities: Not all bridges need to be explicit images. Song Wu used lightweight "implicit" modality samples in Implicit Modality Knowledge Alignment and Uncertainty Estimation to bridge the divide. Meanwhile, Zanwu Liu proposed Modality-Transition Representation Learning (MTRL) in Modality-Transition Representation Learning for Visible-Infrared Person Re-Identification, where the synthetic modality is used only as a loss-function anchor during training, requiring zero extra parameters during inference.
Multi-Step and Bidirectional Transitions
As the field has matured, the "single bridge" has evolved into more complex structures. Mahdi Alehdaghi and Pourya Shamsolmoali proposed Bidirectional Multi-Step Domain Generalization (BMDG) in Bidirectional Multi-Step Domain Generalization for Visible-Infrared Person Re-Identification. Instead of one intermediate step, they create a sequence of virtual domains that gradually mix features from both modalities. By transitioning from both ends toward the middle simultaneously, the model avoids bias toward either the visible or infrared source.
This bidirectional approach is also seen in medical imaging. Huafeng Li developed BSAFusion in BSAFusion: A Bidirectional Stepwise Feature Alignment Network for Unaligned Medical Image Fusion, which uses modality-swapping to inject global information from one source into another, bridging the gap between imaging types like CT and MRI.
Why It Works: Curriculum and Constraints
The success of these intermediate modalities can be attributed to two main factors:
- Optimization Curriculum: Directly minimizing the distance between two disparate modalities is a difficult optimization problem. As noted by Liuxiang Qiu in High-Order Structure Based Middle-Feature Learning, the middle-feature space acts as a learning curriculum. It provides a smoother path for the gradient, preventing the model from getting stuck in poor local minima.
- Structural Preservation: When bridging abstract domains (like hand-drawn sketches) and detailed imagery, as explored by Xingyu Liu in Differentiable Auxiliary Learning for Sketch Re-Identification, the intermediate modality can be designed to retain spatial structure while adopting target textures. This ensures that the model learns to align the content of the images rather than just their statistical distributions.
By shifting the focus from "how to map A to B" to "how to build a common ground between A and B," these researchers have turned a "nightmare" cross-modal gap into a manageable, structured learning task.