Unbiased Prototype Consistency Learning for Multi-Modal and Multi-Task Object Re-Identification

Zhongao Zhou, Bin Yang, Wenke Huang, Jun Chen, Mang YeNeurIPS 2025 · OpenReview

TL;DR

Researchers have developed a unified framework called Unbiased Prototype Consistency Learning (UPCL) that performs object re-identification across different modalities (RGB, Infrared, Thermal) and categories (people and vehicles) simultaneously. By using "modality-unbiased prototypes" and cluster-based regularization, the model overcomes the distribution gaps that usually require separate, resource-heavy models for every specific task.

What Problem Does This Paper Address?

Object Re-Identification (ReID) is the computer vision task of identifying the same person or vehicle across different non-overlapping camera views. While we’ve seen great progress, most existing models are "specialists." You might have one model for RGB person retrieval, another for Thermal-to-RGB vehicle retrieval, and a third for multi-spectral fusion.

This specialization creates two major headaches:

  1. Resource Waste: Deploying a dozen task-specific models in a real-world surveillance system is computationally expensive and hard to manage.
  2. Optimization Conflicts: Cross-modal retrieval (matching a Thermal image to an RGB image) and multi-modal retrieval (fusing RGB+Thermal to find a match) actually pull the feature space in different directions, making it hard for a single model to do both well.
Comparison of traditional limitations and the proposed UPCL method.

To address these issues, the authors introduce the $\rm {M^3T}$-ReID task: a unified approach where one model handles multiple modalities and multiple object categories at once.

Methods

The authors propose the Unbiased Prototype Consistency Learning (UPCL) framework, built on top of a pre-trained CLIP visual encoder. It introduces two novel modules designed to harmonize the feature space across different tasks:

The Unbiased Prototype Consistency Learning (UPCL) framework architecture.
  • Unbiased Prototypes-guided Modality Enhancement (UPME): In ReID, a "prototype" is the average feature vector for a specific identity. Usually, these are biased toward a single modality (like RGB). UPME calculates a "modality-unbiased" prototype by averaging features across RGB, Near-Infrared (NI), and Thermal (TI). This acts as a stable anchor, forcing the model to learn features that are consistent regardless of the camera type.
  • Cluster Prototype Consistency Regularization (CPCR): People and vehicles have very different "discriminative semantics"—you look for clothing and pose for people, but logos and light shapes for cars. CPCR uses a parameter-free clustering algorithm to group identity prototypes into broader category clusters. This helps the model learn stable, category-specific traits without needing manual attribute labels.

Datasets

The study utilizes two primary multi-modal benchmarks, which they combined to create a diverse multi-task environment:

  • RGBNT201: A large-scale multi-modal person ReID dataset containing RGB, Near-Infrared, and Thermal images.
  • RGBNT100: A similar multi-modal dataset, but focused on vehicles.
  • MSVR310: Used specifically for cross-domain evaluation to see how well the model generalizes to entirely new, unseen environments.

Key Insights and Ideas

  • The "Tug-of-War" in Feature Space: The authors identified that cross-modal tasks want to minimize the gap between different modalities of the same object, while multi-modal fusion wants to create a new robust space. UPCL treats the unbiased prototype as a "bridge" that satisfies both requirements.
  • Prototypes as Semantic Anchors: Instead of just using prototypes to identify "Person A," the authors realized that clustering these prototypes reveals higher-level category information. This allows the model to "know" it's looking at a vehicle versus a person and adjust its feature extraction accordingly.
  • Harmonic Mean Matters: To evaluate a "Jack of all trades" model, you can't just look at the best score. The authors use the Harmonic Mean of various tasks, which penalizes the model if it performs brilliantly on one task but fails on another, ensuring true multi-task balance.

Key Results

The UPCL framework significantly outperformed existing state-of-the-art (SOTA) methods that were adapted for this unified task:

  • On RGBNT201 (People): UPCL achieved a 20.75% mAP (mean Average Precision), a massive jump over the previous SOTA (PromptMA) at 15.32%.
  • On RGBNT100 (Vehicles): The lead was even more dramatic, with UPCL hitting 22.63% mAP compared to PromptMA’s 13.93%.
  • Cross-Domain Generalization: When tested on the unseen MSVR310 dataset, UPCL maintained much higher performance than competitors, proving that its learned features are robust and not just overfitted to the training cameras.

Limitations and Future Work

While a major step forward, the authors note a few hurdles:

  • Data Scarcity: There are relatively few high-quality multi-modal datasets for categories beyond pedestrians and vehicles. Expanding this to a "Universal ReID" (including pets, luggage, etc.) is the next frontier.
  • Specialist vs. Generalist Gap: In very narrow, single-task scenarios, a highly tuned specialist model might still slightly edge out a unified model. The authors aim to further close this gap in future iterations.
  • Scaling: Future work will focus on extending UPCL to more generalized multi-modal and multi-task retrieval scenarios beyond the current benchmarks.

Summary updated

Abstract

In object re-identification (ReID) task, both cross-modal and multi-modal retrieval methods have achieved notable progress. However, existing approaches are designed for specific modality and category (person or vehicle) retrieval task, lacking generalizability to others. Acquiring multiple task-specific models would result in wasteful allocation of both training and deployment resources. To address the practical requirements for unified retrieval, we introduce Multi-Modal and Multi-Task object ReID ($\rm {M^3T}$-ReID). The $\rm {M^3T}$-ReID task aims to utilize a unified model to simultaneously achieve retrieval tasks across different modalities and different categories. Specifically, to tackle the challenges of modality distibution divergence and category semantics discrepancy posed in $\rm {M^3T}$-ReID, we design a novel Unbiased Prototype Consistency Learning (UPCL) framework, which consists of two main modules: Unbiased Prototypes-guided Modality Enhancement (UPME) and Cluster Prototype Consistency Regularization (CPCR). UPME leverages modality-unbiased prototypes to simultaneously enhance cross-modal shared features and multi-modal fused features. Additionally, CPCR regulates discriminative semantics learning with category-consistent information through prototypes clustering. Under the collaborative operation of these two modules, our model can simultaneously learn robust cross-modal shared feature and multi-modal fused feature spaces, while also exhibiting strong category-discriminative capabilities. Extensive experiments on multi-modal datasets RGBNT201 and RGBNT100 demonstrates our UPCL framework showcasing exceptional performance for $\rm {M^3T}$-ReID. The code is available at https://github.com/ZhouZhongao/UPCL.
BibTeX
@inproceedings{zhou2025unbiased,
  author = {Zhongao Zhou and Bin Yang and Wenke Huang and Jun Chen and Mang Ye},
  title = {{Unbiased Prototype Consistency Learning for Multi-Modal and Multi-Task Object Re-Identification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=WWa5x1WnEw},
}
Figure