CLIP-Driven Semantic Discovery Network for Visible-Infrared Person Re-Identification

Xiaoyan Yu, Neng Dong, Liehuang Zhu, Hao Peng, Dapeng Tao

Abstract

Visible-infrared person re-identification (VIReID) primarily deals with matching identities across person images from different modalities. Due to the modality gap between visible and infrared images, cross-modality identity matching poses significant challenges. Recognizing that high-level semantics of pedestrian appearance, such as gender, shape, and clothing style, remain consistent across modalities, this paper intends to bridge the modality gap by infusing visual features with high-level semantics. Given the capability of CLIP to sense high-level semantic information corresponding to visual representations, we explore the application of CLIP within the domain of VIReID. Consequently, we propose a CLIP-Driven Semantic Discovery Network (CSDN) that consists of Modality-specific Prompt Learner, Semantic Information Integration (SII), and High-level Semantic Embedding (HSE). Specifically, considering the diversity stemming from modality discrepancies in language descriptions, we devise bimodal learnable text tokens to capture modality-private semantic information for visible and infrared images, respectively. Additionally, acknowledging the complementary nature of semantic details across different modalities, we integrate text features from the bimodal language descriptions to achieve comprehensive semantics. Finally, we establish a connection between the integrated text features and the visual features across modalities. This process embeds rich high-level semantic information into visual representations, thereby promoting the modality invariance of visual representations. The effectiveness and superiority of our proposed CSDN over existing methods have been substantiated through experimental evaluations on multiple widely used benchmarks.

Introduction

Person Re-Identification (ReID) is the task of matching the same individual across different, non-overlapping camera views. While standard ReID works well with color images, real-world 24-hour surveillance relies on infrared (IR) cameras at night. The core challenge is the modality gap: visible images contain rich color and texture, while IR images are monochromatic and capture thermal/shape information. Traditional models struggle to find common ground between these two data types.

However, high-level semantics—such as "a person in a long coat"—remain stable regardless of the camera type. This paper introduces the CLIP-Driven Semantic Discovery Network (CSDN), which leverages the pre-trained knowledge of CLIP (Contrastive Language-Image Pre-training) to use language as a bridge for aligning visual features across modalities.

Methodology

The CSDN framework operates in three distinct training stages to progressively align visual and semantic features.

The CSDN architecture consists of three stages: Modality-specific Prompt Learner, Semantic Information Integration via Attention Fusion, and High-level Semantic Embedding.
  1. Modality-specific Prompt Learner (MsPL): Visible and IR images emphasize different details (e.g., color vs. shape). MsPL utilizes two separate learnable text prompts to capture these modality-private semantic details.
  2. Semantic Information Integration (SII): Recognizing that descriptions are complementary, an Attention Fusion (AF) module treats visible text features as "Query" and IR text features as "Key/Value" to create a single, comprehensive "integrated text feature."
  3. High-level Semantic Embedding (HSE): The model "injects" this rich semantic information into the visual encoders. By forcing visual features to align with integrated text features, the model learns to ignore modality-specific noise and focus on identity-defining characteristics.

During inference, the text components are discarded, and identity matching is performed using only the visual encoders, ensuring computational efficiency.

Experimental Results

The model was evaluated on two primary benchmarks: SYSU-MM01 (large-scale, indoor/outdoor) and RegDB (dual-camera setup).

Comparison of CSDN with state-of-the-art methods on the SYSU-MM01 dataset. CSDN achieves superior performance across all metrics.

As shown in the table above, CSDN achieved state-of-the-art (SOTA) performance:

  • SYSU-MM01: Reached 75.2% Rank-1 accuracy and 71.8% mAP in "all-search" mode, significantly outperforming previous methods like DSCNet (73.8% Rank-1).
  • RegDB: Achieved 89.0% Rank-1 accuracy in visible-to-infrared search mode.
  • Efficiency: Despite using the massive CLIP framework during training, inference time remains practical at approximately 59ms per batch, comparable to standard ResNet-based models.

Conclusion

CSDN successfully bridges the modality gap in VIReID by using CLIP-driven semantics as a universal translator. While current prompts are learnable tokens, future work will explore more advanced multi-modal large models to extract even finer-grained semantic information and strengthen the link between specific image regions and descriptive language.

Summary updated

Abstract

Visible-infrared person re-identification (VIReID) primarily deals with matching identities across person images from different modalities. Due to the modality gap between visible and infrared images, cross-modality identity matching poses significant challenges. Recognizing that high-level semantics of pedestrian appearance, such as gender, shape, and clothing style, remain consistent across modalities, this paper intends to bridge the modality gap by infusing visual features with high-level semantics. Given the capability of CLIP to sense high-level semantic information corresponding to visual representations, we explore the application of CLIP within the domain of VIReID. Consequently, we propose a CLIP-Driven Semantic Discovery Network (CSDN) that consists of Modality-specific Prompt Learner, Semantic Information Integration (SII), and High-level Semantic Embedding (HSE). Specifically, considering the diversity stemming from modality discrepancies in language descriptions, we devise bimodal learnable text tokens to capture modality-private semantic information for visible and infrared images, respectively. Additionally, acknowledging the complementary nature of semantic details across different modalities, we integrate text features from the bimodal language descriptions to achieve comprehensive semantics. Finally, we establish a connection between the integrated text features and the visual features across modalities. This process embed rich high-level semantic information into visual representations, thereby promoting the modality invariance of visual representations. The effectiveness and superiority of our proposed CSDN over existing methods have been substantiated through experimental evaluations on multiple widely used benchmarks. The code will be released at \url{https://github.com/nengdong96/CSDN}.
BibTeX
@article{yu2025clip,
  author = {Xiaoyan Yu and Neng Dong and Liehuang Zhu and Hao Peng and Dapeng Tao},
  title = {{CLIP-Driven Semantic Discovery Network for Visible-Infrared Person Re-Identification}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=sW0m6fZaxb},
}
Figure