Diffusion-based Synthetic Data Generation for Visible-Infrared Person Re-Identification

Wenbo Dai, Lijing Lu, Zhihang LiAAAI 2025 · DOI

TL;DR

This paper introduces DiVE, a novel framework that leverages diffusion models to generate synthetic, identity-preserving Visible-Infrared (VI) person Re-Identification (ReID) data. By decoupling identity and modality representations, DiVE effectively expands existing datasets, leading to significant performance improvements in VI-ReID models.

What Problem Does This Paper Address?

Person Re-Identification (ReID) aims to match individuals across different camera views. Visible-Infrared (VI) ReID extends this to scenarios involving both visible (RGB) and infrared (IR) cameras, crucial for nighttime surveillance. However, training robust VI-ReID models is hampered by the scarcity and high cost of annotating large-scale, paired RGB-IR datasets. Existing methods often struggle with the inherent domain discrepancy between RGB and IR images, and generating realistic, identity-consistent synthetic data for this cross-modal task remains a significant challenge.

Methods

The core of DiVE lies in its innovative use of diffusion models, specifically Stable Diffusion (SD), for synthetic data generation. The framework operates in two key stages:

  1. Identity and Modality Decoupling: DiVE first learns to disentangle identity and modality representations. It uses a technique inspired by Textual Inversion to extract identity embeddings from RGB images, ensuring these embeddings are modality-invariant. Simultaneously, it employs a fine-tuning approach (similar to DreamBooth using LoRA modules) to capture modality-specific characteristics, distinguishing between visible and infrared domains. This decoupling allows for flexible manipulation of identity and modality.
  2. Synthetic Data Generation: Using the learned representations, DiVE constructs text prompts that combine identity and modality information. These prompts are then fed into a fine-tuned Stable Diffusion model to generate synthetic IR images that preserve the identity of the original RGB images. The framework can also handle fine-grained distinctions by incorporating information about different camera views, further enriching the synthetic data.

Datasets

The paper evaluates DiVE on two VI-ReID datasets:

  • SYSU-MM01: Used for both training and evaluation.
  • LLCM: Used for evaluation.

Additionally, they leverage RGB person ReID datasets for expansion:

  • Market-1501
  • CUHK-NP

Key Insights and Ideas

  • Diffusion Models for Cross-Modal Synthesis: The paper demonstrates the power of diffusion models, particularly pre-trained ones like Stable Diffusion, for generating high-fidelity, cross-modal synthetic data in the challenging VI-ReID domain.
  • Decoupling Identity and Modality is Key: The success of DiVE hinges on its ability to separate identity information from modality-specific features. This allows for generating IR images that look like IR but retain the identity of the original RGB image, and vice-versa.
  • Textual Inversion and LoRA for Control: Utilizing techniques like Textual Inversion for identity and LoRA for modality allows for fine-grained control over the generation process, enabling the creation of diverse and realistic synthetic samples.
  • Fine-grained Modality Descriptions Improve Performance: Distinguishing between different camera views within a modality (e.g., IR camera 1 vs. IR camera 2) leads to more precise modality information and better ReID performance.

Key Results

  • Significant Performance Gains: VI-ReID models trained on synthetic data generated by DiVE consistently show notable improvements. For instance, on the SYSU-MM01 dataset, DiVE improved Rank-1 accuracy by 4.37% and mAP by 5.2% in the "All-search" scenario.
  • State-of-the-Art Enhancement: The state-of-the-art method CAJ, when trained with DiVE's synthetic data, achieved an improvement of about 9% in mAP on the LLCM dataset.
  • Qualitative Improvements: Visual comparisons show that DiVE generates more semantically consistent and less artifact-prone IR images compared to GAN-based methods. The generated images maintain identity details like clothing and hairstyles.
  • Effective Dataset Expansion: DiVE effectively expands existing datasets, demonstrating that synthetic data can significantly boost performance even when trained on external visible-based datasets.

Limitations and Future Work

  • Optimal ID Count: The paper notes that there's an optimal number of augmented IDs. Too few IDs may not provide sufficient diversity, while too many can harm performance, possibly due to the long-tail distribution of datasets and challenges in extracting ID features from very few images. Finding this balance is crucial.
  • Future Directions: The authors suggest that DiVE's ability to generate diverse, identity-preserving synthetic data opens new avenues for bridging generative models and person ReID research. Further exploration into optimizing the balance of augmented IDs and refining modality distinctions could be promising.

Summary updated

BibTeX
@inproceedings{dai2025diffusion,
  author = {Wenbo Dai and Lijing Lu and Zhihang Li},
  title = {{Diffusion-based Synthetic Data Generation for Visible-Infrared Person Re-Identification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i11.33216},
}
Figure