TL;DR
This paper introduces GSMEN, a novel network for Visible-Infrared Person Re-Identification (VI Re-ID) that tackles the significant modality discrepancy between visible and infrared images. By incorporating contour information and a new graph-based sampling strategy, GSMEN achieves state-of-the-art performance, improving the ability to identify individuals across different camera types and lighting conditions.
What Problem Does This Paper Address?
Person re-identification (Re-ID) is crucial for surveillance and security, aiming to match individuals across non-overlapping camera views. However, traditional Re-ID struggles with varying lighting and camera conditions. Visible-Infrared (VI) Re-ID extends this by matching images from visible light cameras with those from infrared cameras, which are useful in low-light or nighttime scenarios. The primary challenge in VI Re-ID is the substantial difference in appearance between visible and infrared images (modality discrepancy). Existing methods often rely on complex modal transformations or shared feature learning, but they tend to overlook the importance of person contour information, which is surprisingly invariant across modalities and can aid in matching. Furthermore, current training strategies often use simple sampling methods that fail to effectively capture the subtle differences between challenging "hard" samples, hindering discriminative learning.
Methods
The proposed GSMEN network introduces two key innovations:
- Contour Expansion Module (CEM): This module leverages the modality-invariant nature of person contours. It first extracts contour maps from both visible and infrared images using a pre-trained human parsing network. These contour features are then fused with the original image features. The paper explores different fusion methods (element-wise addition and concatenation) and finds that element-wise concatenation, which augments local features with global contour features, yields the best results. This fusion aims to reduce modality discrepancy and enhance identity representation.
- Cross-modality Graph Sampler (CGS): To address the limitations of random or uniform sampling, CGS constructs a graph that represents relationships between classes in both visible and infrared modalities. At the beginning of each training epoch, it calculates class-wise distances using features from the latest trained model. Then, for each mini-batch, it selects an "anchor" class and its top-k nearest neighboring classes. This strategy ensures that samples within a batch are mostly similar, creating challenging "hard" sample pairs that help the model learn to distinguish between similar individuals and improve its discriminative power.
The overall network uses a two-stream convolutional neural network architecture with ResNet-50 as the backbone, enhanced with Non-local Attention for better feature extraction. The loss function combines triplet loss, cross-entropy loss, and a self-supervised Barlow Twins loss for improved performance.
Datasets
The proposed method is evaluated on two widely used VI Re-ID datasets:
- SYSU-MM01: A large-scale dataset with 491 identities captured by four visible and two infrared cameras. It includes two search modes: All-Search and Indoor-Search. The dataset comprises training images and test sets with infrared images as queries.
- RegDB: A dataset containing images of 412 identities, with each identity having 10 visible and 10 infrared images. It offers two testing protocols: Infrared-to-Visible (IR-to-VIS) and Visible-to-Infrared (VIS-to-IR).
Key Insights and Ideas
- Contour invariance is a powerful cue: The paper highlights that contour information, often overlooked, is remarkably stable across visible and infrared modalities. This invariance makes it a valuable feature for bridging the gap between the two.
- Strategic sampling matters: Simple sampling methods are insufficient for VI Re-ID. The CGS sampler's ability to group similar classes into batches is crucial for exposing the model to challenging cross-modal pairs, leading to better discriminative learning.
- Synergistic effect of CEM and CGS: The ablation studies show that while both CEM and CGS individually improve performance, their combined use leads to the most significant gains, demonstrating their complementary roles in enhancing feature representation and optimizing the training process.
Key Results
The GSMEN network achieves state-of-the-art performance on both SYSU-MM01 and RegDB datasets.
- On the RegDB dataset, for the VIS→IR task, GSMEN achieves a Rank-1 accuracy of 93.69% and an mAP of 92.56%.
- On the SYSU-MM01 dataset, in the All-Search mode, GSMEN achieves a Rank-1 accuracy of 72.97% and an mAP of 70.54%. In the Indoor-Search mode, it achieves 78.22% Rank-1 and 80.13% mAP.
- These results significantly outperform existing state-of-the-art methods, including GLMC and AGMNet, by substantial margins in various evaluation settings.
- Qualitative analysis shows that GSMEN correctly retrieves more relevant images, especially in challenging rearview scenarios, where other methods might make erroneous matches.
Limitations and Future Work
- Contour Detector Choice: While the paper demonstrates that SCHP is superior to Canny edge detection for contour extraction, the choice of contour detector can still impact performance. Future work could explore more advanced or adaptive contour detection methods.
- Fusion Method Exploration: Although element-wise concatenation was chosen for fusion, the paper briefly mentions other fusion methods. Further investigation into optimal fusion strategies could yield additional improvements.
- Computational Cost: The introduction of contour extraction and graph sampling might increase the computational overhead. Optimizing these components for efficiency could be a future direction.
The paper concludes that the proposed GSMEN, with its novel CEM and CGS modules, offers a promising direction for robust and efficient VI Re-ID, with potential applications in various real-world scenarios.
Summary updated