TL;DR
This paper introduces Mix-Modality Person Re-Identification (MM-ReID), a more realistic paradigm where both query and gallery sets contain a mixture of visible and infrared images. To combat the "modality confusion" that plagues existing models in this setting, the authors propose a new loss function (CIDHL) to harmonize feature distributions and a similarity optimization strategy (MBSOS) that uses "bridge samples" to navigate the modality gap during retrieval.
What Problem Does This Paper Address?
Person Re-Identification (ReID) is the task of identifying the same individual across different cameras. While standard ReID uses visible light (RGB) images, Visible-Infrared ReID (VI-ReID) was developed to handle low-light conditions by matching RGB images against infrared (IR) captures.
However, the authors argue that the current VI-ReID research is stuck in a "bi-modality mutual retrieval" bubble: it assumes you are either searching for an IR image in an RGB gallery or vice versa. In the real world, a surveillance database (the gallery) likely contains a mix of both types of images captured at different times of day. When existing models are tested in this "mix-modality" environment, they suffer from Modality Confusion. Because images of the same modality (e.g., two different people both in IR) often look more similar to a computer than images of the same person in different modalities, the model incorrectly prioritizes modality over identity.
Methods
The authors propose two primary solutions to solve modality confusion:
-
Cross-Identity Discrimination Harmonization Loss (CIDHL): Standard triplet losses compare individual samples, which can be noisy. CIDHL instead works with identity centers—the average feature representation of a person within a specific modality. It forces the centers of the same identity (across different modalities) to move closer together while pushing centers of different identities further apart. Crucially, it also includes a "harmonization" component that aggregates samples of the same modality and identity toward their respective centers, sharpening the model's ability to distinguish between people even when they appear in the same camera mode.
-
Modality Bridge Similarity Optimization Strategy (MBSOS): This is a post-processing technique used during the testing/retrieval phase. Instead of calculating a direct (and often inaccurate) distance between a visible query and an infrared gallery target, MBSOS treats retrieval as a pathfinding problem. It looks for "bridge samples" in the gallery that share the same modality as the query. By calculating the distance through these intermediate "stepping stones," the system reduces the distortive effect of the modality gap, leading to more accurate matches.
Datasets
The authors evaluated their methods on three standard VI-ReID benchmarks, adapting them into the new MM-ReID testing paradigm by creating various "mixing ratios" of visible and infrared images:
- SYSU-MM01: A large-scale dataset with over 300,000 images.
- RegDB: A dual-camera dataset with 412 identities.
- LLCM: Currently the largest dataset for low-light cross-modality ReID.
Key Insights and Ideas
- The "Modality First" Trap: The paper highlights that in mixed settings, models often learn that "modality" is a stronger feature than "identity." This is why a model might perform well on pure cross-modality tasks but fail when the gallery is mixed.
- Centers over Samples: By using identity centers in the loss function, the authors provide a more stable training signal that is less susceptible to outliers or mislabeled data.
- Stepping Stones: The MBSOS strategy is a clever realization that the shortest path between two points in a feature space isn't always a straight line if that line has to cross a massive "modality canyon." Using same-modality neighbors as bridges provides a more reliable similarity metric.
Key Results
The authors integrated CIDHL and MBSOS into several state-of-the-art (SOTA) models like AGW, HCTL, and DEEN. The improvements were significant across the board:
- SYSU-MM01: On the AGW baseline, adding the proposed methods boosted Rank-1 accuracy by 18.86% and mAP (mean Average Precision) by 13.76%.
- RegDB: The AGW model saw a Rank-1 improvement of 6.09% and an mAP jump of 8.84%.
- Consistency: The methods proved "transferable," meaning they generally improved the performance of any base architecture they were added to, confirming that modality confusion is a universal hurdle in MM-ReID.
Limitations and Future Work
The authors identify two main areas for improvement:
- Data Utilization: The current training setup uses a two-branch model that inputs modalities separately. They suggest that a more integrated way of utilizing mixed data during the training phase itself could yield better results.
- Bridge Sophistication: The current MBSOS strategy uses a relatively simple "average" approach for bridge samples. Future work could involve exploring more complex potential relationships between same-modality and cross-modality sample distances to further refine the pathfinding logic.
Summary updated