Video-Level Language-Driven Video-Based Visible-Infrared Person Re-Identification

Shuang Li, Jiaxu Leng, Changjiang Kuang, Mingpi Tan, Xinbo Gao

Abstract

Visible-Infrared Person Re-Identification (VI-ReID) is a critical task for 24-hour surveillance systems, requiring the matching of pedestrian identities across visible (RGB) and near-infrared (IR) video sequences. This task is complicated by the significant modality gap between spectra and the need to capture temporal dynamics. This paper presents the Video-level Language-driven (VLD) framework, which leverages the Contrastive Language-Image Pre-training (CLIP) model as a semantic bridge. By introducing lightweight spatiotemporal prompting mechanisms, the VLD framework achieves state-of-the-art performance with significantly reduced computational overhead compared to existing methods.

Introduction

In modern security infrastructure, cameras often switch from visible light sensors during the day to infrared sensors at night. This transition creates a fundamental challenge for automated person re-identification: the appearance of a person changes drastically between modalities. While image-based VI-ReID has seen significant progress, video-based VI-ReID—which utilizes temporal information like gait and motion—remains difficult due to the lack of effective cross-modal alignment strategies.

Traditional approaches often rely on adversarial learning or generative models to bridge the modality gap. However, these methods frequently struggle to capture fine-grained semantic details and often ignore the rich temporal context available in video tracklets. Furthermore, while large-scale models like CLIP offer powerful cross-modal alignment capabilities, they are typically trained on static RGB images, making them inherently "blind" to infrared data and temporal sequence dynamics.

Methodology

The proposed VLD framework addresses these limitations through two primary innovations designed to adapt CLIP for the video-based infrared-visible domain: Invariant-Modality Language Prompting (IMLP) and Spatial-Temporal Prompting (STP).

Invariant-Modality Language Prompting (IMLP)

The core insight of IMLP is that while a person's visual representation (pixels) changes between RGB and IR, their semantic description remains constant. A "person in a long coat" is a consistent description regardless of the sensor type. IMLP utilizes learnable text prompts—such as "A [X] person observed in both day and night conditions"—to establish a modality-shared semantic space. By jointly fine-tuning the visual encoder and the prompt learner, the model learns to map both RGB and IR video features to identical identity-specific text prototypes, using language as a modality-invariant anchor.

Spatial-Temporal Prompting (STP)

To incorporate temporal awareness into the Vision Transformer (ViT) backbone of CLIP, the authors developed the STP module. Unlike traditional methods that append temporal layers (like LSTMs) after the feature extraction phase, STP operates within the intermediate layers of the ViT. It consists of two components:

  1. Spatial-Temporal Hub (STH): A learnable tensor that acts as a central exchange, aggregating spatial information from individual frames and diffusing it across the entire sequence through a specialized attention mechanism.
  2. Spatial-Temporal Aggregation (STA): An attention module that ensures the STH focuses on identity-relevant features, such as body shape or clothing patterns, while suppressing background noise.

Experimental Results

The VLD framework was evaluated on two benchmark datasets: HITSZ-VCM and BUPTCampus.

Performance Metrics

On the HITSZ-VCM dataset, the framework achieved a 74.3% Rank-1 accuracy in the Infrared-to-Visible setting, representing a 5.1% improvement over the previous state-of-the-art (SAADG). In the Visible-to-Infrared direction, the improvement was even more pronounced, with a 10.4% increase in Rank-1 accuracy. Similar gains were observed on the larger BUPTCampus dataset, where VLD significantly outperformed existing models like AuxNet in both Rank-1 and mean Average Precision (mAP).

Efficiency and Generalization

A key advantage of the VLD framework is its efficiency. By utilizing a prompting-based approach rather than heavy architectural modifications, the model adds only 2.39M parameters. This allows the system to be trained in approximately 2 hours on a single A6000 GPU—an 80% reduction in training time compared to previous leading models like MITML. Furthermore, cross-dataset testing demonstrated that VLD possesses superior robustness, suggesting that the language-driven approach helps the model learn generalizable identity features rather than dataset-specific artifacts.

Conclusion

The VLD framework demonstrates that language can serve as an effective "universal bridge" for cross-modal video matching. By integrating lightweight spatiotemporal prompts directly into the transformer architecture, the model achieves superior accuracy and efficiency. Future work will focus on adapting the framework for semi-supervised or unsupervised scenarios to reduce the reliance on large-scale labeled datasets, as well as improving handling for variable-length pedestrian trajectories.

Summary updated

BibTeX
@article{li2025video,
  author = {Shuang Li and Jiaxu Leng and Changjiang Kuang and Mingpi Tan and Xinbo Gao},
  title = {{Video-Level Language-Driven Video-Based Visible-Infrared Person Re-Identification}},
  journal = {IEEE Trans. Inf. Forensics Secur.},
  year = {2025},
  doi = {10.1109/tifs.2025.3573672},
}
Figure