TL;DR
This paper introduces a novel weakly supervised framework for visible-infrared person re-identification (VIReID) that completely eliminates the need for expensive cross-modal identity annotations. By training modality-specific "heterogeneous experts" and forcing them to collaboratively align their predictions, the method achieves performance that rivals fully supervised approaches using only single-modality labels.
What Problem Does This Paper Address?
Visible-Infrared Person Re-Identification (VIReID) is a critical task for 24/7 intelligent surveillance, requiring a system to match a pedestrian's image captured by a visible (RGB) camera during the day with an image captured by an infrared (IR) camera at night.
The core bottleneck in training these models is the modality gap and annotation cost. Because visible and infrared cameras often operate asynchronously (e.g., day vs. night), it is incredibly difficult and expensive to manually link identities across modalities.
Existing unsupervised methods try to cluster data without any labels but suffer from massive pseudo-label noise due to the drastic visual differences between RGB and IR images. Fully supervised methods work well but demand extensive cross-modal annotations. The authors tackle this dilemma by proposing a weakly supervised setting: assuming you have identity labels within the visible dataset and within the infrared dataset, but absolutely no cross-modal links telling you which visible ID matches which infrared ID.
Methods
The authors introduce a Heterogeneous Expert Collaborative Consistency Learning framework, which operates in three main stages:
- Heterogeneous Expert Learning (HEL): Separate feature encoders and classification networks are trained independently for each modality. The visible network becomes an expert at RGB features, while the infrared network becomes an expert at thermal features.
- Cross-Modal Relationship Establishment (CRE): To bridge the modalities, the visible expert is used to predict the identities of infrared samples, and vice versa. The framework fuses these cross-modality predictions and categorizes the relationships into three buckets: consistent matches (both experts agree), unique matches (one expert is confident), and conflicting matches (experts disagree).
- Collaborative Consistency Learning (CCL): Using these discovered relationships, a shared classifier ($W^c$) is trained. Strong cross-entropy constraints are applied to consistent matches, while a relaxed, noise-tolerant loss function ($L_{id}^{weak}$) is used for conflicting matches to extract value without injecting noise. Simultaneously, identity "prototypes" are maintained to dynamically align the experts' decision boundaries, ensuring they don't forget their single-modality knowledge while learning to agree across modalities.
Datasets
The method was evaluated on two prominent, large-scale benchmarks:
- SYSU-MM01: The standard benchmark for VIReID, containing 491 pedestrian identities with 287,628 visible images and 15,792 infrared images. Evaluation is performed under both "all-search" and "indoor-search" modes.
- LLCM: A highly challenging, low-light cross-modal dataset containing 1,064 identities captured by 9 visible and 9 infrared cameras in low-light environments.
Key Insights and Ideas
- Complementary Modality Experts: Instead of trying to force a single model to bridge the RGB-IR gap from day one, the authors embrace the gap. By training separate experts, the IR expert learns to rely on structural/geometric cues (since color is missing), while the visible expert leverages rich textures. Their combined cross-modal predictions are far more robust than a single model's pseudo-labels.
- Exploiting Contradictions Safely: Most pseudo-labeling frameworks discard data when predictions conflict. Here, the authors design a relaxed loss that allows the model to learn from "conflicting matches" by safely penalizing only the highly improbable matches, preventing label noise from degrading the shared encoder.
- Entropy-Driven Homogeneity: The framework uses information entropy to dynamically balance collaborative learning. If an expert's prediction is highly uncertain (high entropy), the consistency constraint is tightened, forcing it to learn from its peer.
Key Results
The weakly supervised method significantly outperforms existing semi-supervised and unsupervised state-of-the-art (SOTA) methods:
- SYSU-MM01 (All Search): Achieves 70.4% Rank-1 accuracy and 66.6% mAP. This outperforms the leading semi-supervised method (DPIS) by 12.0% in Rank-1 accuracy and beats unsupervised methods (like GUR) by 6.9%.
- LLCM (Visible to IR): Achieves 55.3% Rank-1 accuracy and 58.7% mAP, outperforming unsupervised graph-matching techniques (PGM) by 10.4% in Rank-1.
Feature visualizations show that the collaborative consistency learning successfully pushes inter-class features apart while pulling cross-modal intra-class features into tight, distinct clusters.
Limitations and Future Work
- Intra-Modality Label Perfection: The framework assumes that the single-modality labels provided for initial expert training are perfectly clean. In real-world web-scraped or automatically tracked data, intra-modality label noise might exist, which could degrade the experts' initial capabilities.
- Future Directions: The authors note that future work will focus on refining adaptive learning mechanisms and improving expert collaboration to handle highly ambiguous, occluded, or complex pedestrian poses.
Summary updated