CLIP-Driven Semantic Discovery Network for Visible-Infrared Person Re-Identification

Xiaoyan Yu, Neng Dong, Liehuang Zhu, Hao Peng, Dapeng Tao

TL;DR

Researchers have developed the CLIP-Driven Semantic Discovery Network (CSDN) to solve the "modality gap" in person re-identification (ReID) between visible (RGB) and infrared (IR) cameras. By using the CLIP model to generate and fuse modality-specific text descriptions, the system learns high-level, modality-invariant features (like "clothing style" or "body shape") that allow for highly accurate person matching across day and night surveillance feeds.

What Problem Does This Paper Address?

Person Re-Identification (ReID) is the task of matching the same individual across different, non-overlapping camera views. While standard ReID works well with color images, real-world 24-hour surveillance relies on infrared (IR) cameras at night.

The core challenge is the modality gap: visible images contain rich color and texture, while IR images are monochromatic and capture thermal/shape information. Traditional models struggle to find common ground between these two data types. However, humans can easily recognize that a "person in a long coat" is the same individual regardless of the camera type. This paper seeks to automate that human-like intuition by using high-level semantics—language—as a bridge to align visual features across modalities.

Methods

The authors propose the CLIP-Driven Semantic Discovery Network (CSDN), which leverages the pre-trained knowledge of CLIP (Contrastive Language-Image Pre-training). The training happens in three distinct stages:

  1. Modality-specific Prompt Learner (MsPL): Instead of using a single description for both modalities, the authors argue that visible and IR images emphasize different details (e.g., color vs. shape). They design two separate learnable text prompts to capture these modality-private details.
  2. Semantic Information Integration (SII): Recognizing that these descriptions are complementary, they use an Attention Fusion (AF) module. This module treats visible text features as "Query" and IR text features as "Key/Value" to create a single, comprehensive "integrated text feature" that represents the person's identity across both worlds.
  3. High-level Semantic Embedding (HSE): Finally, the model "injects" this rich semantic information into the visual encoders. By forcing the visual features to align with the integrated text features, the model learns to ignore modality-specific noise (like color shifts) and focus on identity-defining characteristics.

Crucially, during the inference (testing) phase, the text components are discarded. The system performs identity matching using only the visual encoders, maintaining computational efficiency.

Datasets

The model was evaluated on two primary benchmarks for Visible-Infrared ReID:

  • SYSU-MM01: The largest and most challenging dataset, featuring over 280,000 visible images and 15,000 IR images from both indoor and outdoor environments.
  • RegDB: A smaller-scale dataset where each of the 412 identities has 10 visible and 10 infrared images, all collected from a single dual-camera setup.

Key Insights and Ideas

  • Text as a Universal Translator: The most powerful insight is that while image features vary wildly between RGB and IR, the textual description of a person remains stable. By using CLIP as a "semantic anchor," the model can find a common latent space for both modalities.
  • The Necessity of Modality-Specific Prompts: The authors found that a "one-size-fits-all" text prompt is insufficient. Because IR cameras lose color information, the model needs to learn to look for different semantic cues (like silhouette or gait) in IR compared to RGB.
  • Complementary Fusion: By fusing the text features, the model compensates for the weaknesses of each modality. The visible modality provides color/texture, while the IR modality provides robust shape information that isn't affected by lighting conditions.

Key Results

The CSDN achieved state-of-the-art (SOTA) performance on both major benchmarks:

  • SYSU-MM01: Reached a 75.2% Rank-1 accuracy and 71.8% mAP (mean Average Precision) in the "all-search" mode. This is a significant jump over previous non-generative methods like DSCNet (73.8% Rank-1).
  • RegDB: Achieved 89.0% Rank-1 accuracy in the visible-to-infrared search mode, outperforming the previous SOTA (TSME) by 1.7%.
  • Efficiency: Despite using the massive CLIP framework during training, the inference time remains comparable to standard ResNet-based models (~59ms per batch), proving it is practical for real-world deployment.

Limitations and Future Work

The authors acknowledge that the language descriptions generated by the current "prompt learner" paradigm are still relatively "coarse." They rely on a limited number of learnable word tokens rather than rich, descriptive natural language.

Future Work: The team aims to explore more advanced multi-modal large models to extract even finer-grained semantic information. They also suggest that enriching the link between specific image regions and specific words (e.g., "red" linked to "shirt") could further improve the robustness of the visual representations.

Summary updated

Abstract

Visible-infrared person re-identification (VIReID) primarily deals with matching identities across person images from different modalities. Due to the modality gap between visible and infrared images, cross-modality identity matching poses significant challenges. Recognizing that high-level semantics of pedestrian appearance, such as gender, shape, and clothing style, remain consistent across modalities, this paper intends to bridge the modality gap by infusing visual features with high-level semantics. Given the capability of Contrastive Language-Image Pre-training (CLIP) to sense high-level semantic information corresponding to visual representations, we explore the application of CLIP within the domain of VIReID. Consequently, we propose a CLIP-Driven Semantic Discovery Network (CSDN) that consists of Modality-specific Prompt Learner, Semantic Information Integration (SII), and High-level Semantic Embedding (HSE). Specifically, considering the diversity stemming from modality discrepancies in language descriptions, we devise bimodal learnable text tokens to capture modality-private semantic information for visible and infrared images, respectively. Additionally, acknowledging the complementary nature of semantic details across different modalities, we integrate text features from the bimodal language descriptions to achieve comprehensive semantics. Finally, we establish a connection between the integrated text features and the visual features across modalities. This process embed rich high-level semantic information into visual representations, thereby promoting the modality invariance of visual representations. The effectiveness and superiority of our proposed CSDN over existing methods have been substantiated through experimental evaluations on multiple widely used benchmarks.
BibTeX
@article{yu2025clip,
  author = {Xiaoyan Yu and Neng Dong and Liehuang Zhu and Hao Peng and Dapeng Tao},
  title = {{CLIP-Driven Semantic Discovery Network for Visible-Infrared Person Re-Identification}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=sW0m6fZaxb},
}

Loading related content...

Figure