Weakly Supervised Visible-Infrared Person Re-Identification via Heterogeneous Expert Collaborative Consistency Learning

Yafei Zhang, Lingqi Kong, Huafeng Li, Jie Wen

TL;DR

This paper introduces a novel weakly supervised framework for visible-infrared person re-identification (VIReID) that completely eliminates the need for expensive cross-modal identity annotations. By training modality-specific "heterogeneous experts" and forcing them to collaboratively align their predictions, the method achieves performance that rivals fully supervised approaches using only single-modality labels.

What Problem Does This Paper Address?

Visible-Infrared Person Re-Identification (VIReID) is a critical task for 24/7 intelligent surveillance, requiring a system to match a pedestrian's image captured by a visible (RGB) camera during the day with an image captured by an infrared (IR) camera at night.

The core bottleneck in training these models is the modality gap and annotation cost. Because visible and infrared cameras often operate asynchronously (e.g., day vs. night), it is incredibly difficult and expensive to manually link identities across modalities.

Existing unsupervised methods try to cluster data without any labels but suffer from massive pseudo-label noise due to the drastic visual differences between RGB and IR images. Fully supervised methods work well but demand extensive cross-modal annotations. The authors tackle this dilemma by proposing a weakly supervised setting: assuming you have identity labels within the visible dataset and within the infrared dataset, but absolutely no cross-modal links telling you which visible ID matches which infrared ID.

Figure 1. VIReID faces two key challenges: asynchronous visible- infrared (VIS-IR) camera operation causes missing cross-modal identity correspondences, while large appearance differences raise annotation costs beyond those in single-modality scenarios.

Methods

The authors introduce a Heterogeneous Expert Collaborative Consistency Learning framework, which operates in three main stages:

  1. Heterogeneous Expert Learning (HEL): Separate feature encoders and classification networks are trained independently for each modality. The visible network becomes an expert at RGB features, while the infrared network becomes an expert at thermal features.
  2. Cross-Modal Relationship Establishment (CRE): To bridge the modalities, the visible expert is used to predict the identities of infrared samples, and vice versa. The framework fuses these cross-modality predictions and categorizes the relationships into three buckets: consistent matches (both experts agree), unique matches (one expert is confident), and conflicting matches (experts disagree).
  3. Collaborative Consistency Learning (CCL): Using these discovered relationships, a shared classifier ($W^c$) is trained. Strong cross-entropy constraints are applied to consistent matches, while a relaxed, noise-tolerant loss function ($L_{id}^{weak}$) is used for conflicting matches to extract value without injecting noise. Simultaneously, identity "prototypes" are maintained to dynamically align the experts' decision boundaries, ensuring they don't forget their single-modality knowledge while learning to agree across modalities.
Figure 2. Overview of the proposed method. HEL first performs intra-modality training to obtain experts for each modality, generating cross-modality prediction results for CRE. CRE then establishes robust cross-modality correspondences by fusing predictions from multiple experts. CCL utilizes the correspondences provided by CRE to facilitate CMCL and CLAE.

Datasets

The method was evaluated on two prominent, large-scale benchmarks:

  • SYSU-MM01: The standard benchmark for VIReID, containing 491 pedestrian identities with 287,628 visible images and 15,792 infrared images. Evaluation is performed under both "all-search" and "indoor-search" modes.
  • LLCM: A highly challenging, low-light cross-modal dataset containing 1,064 identities captured by 9 visible and 9 infrared cameras in low-light environments.

Key Insights and Ideas

  • Complementary Modality Experts: Instead of trying to force a single model to bridge the RGB-IR gap from day one, the authors embrace the gap. By training separate experts, the IR expert learns to rely on structural/geometric cues (since color is missing), while the visible expert leverages rich textures. Their combined cross-modal predictions are far more robust than a single model's pseudo-labels.
  • Exploiting Contradictions Safely: Most pseudo-labeling frameworks discard data when predictions conflict. Here, the authors design a relaxed loss that allows the model to learn from "conflicting matches" by safely penalizing only the highly improbable matches, preventing label noise from degrading the shared encoder.
  • Entropy-Driven Homogeneity: The framework uses information entropy to dynamically balance collaborative learning. If an expert's prediction is highly uncertain (high entropy), the consistency constraint is tightened, forcing it to learn from its peer.

Key Results

The weakly supervised method significantly outperforms existing semi-supervised and unsupervised state-of-the-art (SOTA) methods:

  • SYSU-MM01 (All Search): Achieves 70.4% Rank-1 accuracy and 66.6% mAP. This outperforms the leading semi-supervised method (DPIS) by 12.0% in Rank-1 accuracy and beats unsupervised methods (like GUR) by 6.9%.
  • LLCM (Visible to IR): Achieves 55.3% Rank-1 accuracy and 58.7% mAP, outperforming unsupervised graph-matching techniques (PGM) by 10.4% in Rank-1.

Feature visualizations show that the collaborative consistency learning successfully pushes inter-class features apart while pulling cross-modal intra-class features into tight, distinct clusters.

Figure 3. Visualization comparison of feature distributions be- tween baseline and our method on SYSU-MM01: intra- and inter- class distances of cross-modal sample features in the first row, and t-SNE mapped 2D feature space visualization in the second row, with 10 randomly selected identities and 10 samples each.

Limitations and Future Work

  • Intra-Modality Label Perfection: The framework assumes that the single-modality labels provided for initial expert training are perfectly clean. In real-world web-scraped or automatically tracked data, intra-modality label noise might exist, which could degrade the experts' initial capabilities.
  • Future Directions: The authors note that future work will focus on refining adaptive learning mechanisms and improving expert collaboration to handle highly ambiguous, occluded, or complex pedestrian poses.

Summary updated

Abstract

To reduce the reliance of visible-infrared person re-identification (ReID) models on labeled cross-modal samples, this paper explores a weakly supervised cross-modal person ReID method that uses only single-modal sample identity labels, addressing scenarios where cross-modal identity labels are unavailable. To mitigate the impact of missing cross-modal labels on model performance, we propose a heterogeneous expert collaborative consistency learning framework, designed to establish robust cross-modal identity correspondences in a weakly supervised manner. This framework leverages labeled data from each modality to independently train dedicated classification experts. To associate cross-modal samples, these classification experts act as heterogeneous predictors, predicting the identities of samples from the other modality. To improve prediction accuracy, we design a cross-modal relationship fusion mechanism that effectively integrates predictions from different experts. Under the implicit supervision provided by cross-modal identity correspondences, collaborative and consistent learning among the experts is encouraged, significantly enhancing the model's ability to extract modality-invariant features and improve cross-modal identity recognition. Experimental results on two challenging datasets validate the effectiveness of the proposed method.
BibTeX
@inproceedings{zhang2025weakly,
  author = {Yafei Zhang and Lingqi Kong and Huafeng Li and Jie Wen},
  title = {{Weakly Supervised Visible-Infrared Person Re-Identification via Heterogeneous Expert Collaborative Consistency Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}

Loading related content...

Figure