Abstract
Visible-Infrared Person Re-identification (VI ReID) aims to match visible and infrared images of the same pedestrians across non-overlapped camera views. These two input modalities contain both invariant information, such as shape, and modality-specific details, such as color. An ideal model should utilize valuable information from both modalities during training for enhanced representational capability. However, the gap caused by modality-specific information poses substantial challenges for the VI ReID model to handle distinct modality inputs simultaneously. To address this, we introduce the Modality-aware and Instance-aware Visual Prompts (MIP) network, designed to effectively utilize both invariant and specific information for identification. Built on a transformer architecture, the MIP model employs modality-specific prompts to adapt to different modality inputs and instance-specific prompts to capture identity-level discriminative clues. Extensive experiments on SYSU-MM01 and RegDB datasets demonstrate that MIP outperforms most state-of-the-art methods.
Introduction: The Modality Gap Challenge
Visible-Infrared Person Re-identification (VI ReID) is a critical task in surveillance, requiring the matching of pedestrian images captured by standard visible-light cameras during the day with those from infrared cameras at night. The primary obstacle is the modality gap: visible images provide rich color and texture, whereas infrared images capture thermal radiation, resulting in significant visual discrepancies.
Traditionally, researchers have attempted to bridge this gap by focusing solely on modality-invariant features, such as body shape, while discarding modality-specific information like color as noise. However, this approach ignores the fact that modality-specific details are often correlated; for example, the material of a garment affects both its visual texture and its thermal signature. The challenge lies in effectively utilizing these disparate clues without the model becoming confused by sensor-specific differences.
The MIP Framework
To address these challenges, the Modality-aware and Instance-aware Visual Prompts (MIP) network was developed. Built upon a Vision Transformer (ViT) backbone, the MIP framework introduces flexibility through two primary modules:
1. Modality-aware Prompt Learning (MPL)
MPL maintains a library of learnable prompt vectors for both visible and infrared modalities. When an image is processed, the model selects the corresponding prompt to inject into the Transformer layers. This mechanism acts as a signal, guiding the global backbone to adapt its processing strategy based on the specific sensor type, thereby reducing interference from the modality gap.
2. Instance-aware Prompt Generator (IPG)
Beyond sensor-level adaptation, the model must distinguish between individuals. The IPG uses a mini-transformer layer to generate customized prompts for each pedestrian instance. These prompts allow the model to dynamically capture identity-level clues—such as a specific bag or gait—that might otherwise be lost in a generic feature extraction process.
To ensure these prompts remain discriminative, the authors implemented an Instance-aware Enhancement Loss (IAEL). This loss function prevents the generated prompts from becoming trivial, forcing them to capture features unique to the specific individual.
Key Insights: From Invariance to Correspondence
A central insight of this work is that modality-specific information should be treated as a feature rather than noise. By being modality-aware, the model can learn the implicit relationships between visible textures and infrared signatures.
As illustrated in the figure above, while standard backbones focus on explicit correspondence (modality-invariant attributes like shape), the MIP approach enables the model to capture implicit correspondences by leveraging modality-specific attributes and semantic information. This dynamic adaptation, inspired by prompting techniques in Large Language Models, allows a static backbone to remain flexible across different environmental conditions.
Experimental Results
The MIP network was evaluated on two major benchmarks, achieving state-of-the-art (SOTA) performance:
- SYSU-MM01: The model achieved a 70.84% Rank-1 accuracy and 66.41% mAP in the "All-Search" mode, a significant improvement over previous CNN and Transformer-based methods.
- RegDB: In the "Infrared to Visible" mode, the model reached 92.38% Rank-1 accuracy.
Qualitative analysis further supported these results. Attention map visualizations indicated that while baseline models often struggled with features that changed drastically between modalities (like a patterned shirt), the MIP model successfully maintained focus on the entire person by utilizing modality-specific prompts.
Conclusion and Future Directions
The MIP framework demonstrates that utilizing modality-specific information through visual prompts is a powerful strategy for VI ReID. By moving away from simple feature filtering toward active modality and instance adaptation, the model achieves superior identification accuracy.
Future work may involve integrating more rigid structural modeling of the human body to further improve performance on smaller datasets like RegDB. Additionally, optimizing the Instance-aware Prompt Generator for greater computational efficiency could facilitate the deployment of this technology in real-time edge computing environments.
Summary updated