Contextualized Multimodal Lifelong Person Re-Identification in Hybrid Clothing States

Robert Long, Rongxin Jiang, Mingrui Yan

TL;DR

This paper introduces CMLReID, a multimodal framework designed for the "LReID-Hybrid" task: identifying people across surveillance cameras while they change clothes and as the model learns from a continuous stream of new data. By leveraging CLIP and dynamic semantic prompting, the model learns to distinguish between "same-cloth" and "cloth-changing" scenarios, significantly reducing catastrophic forgetting and improving accuracy over existing state-of-the-art methods.

What Problem Does This Paper Address?

Person Re-Identification (ReID) is the task of spotting the same individual across different camera views. Historically, researchers focused on two distinct flavors:

  1. Same-Cloth (SC): Assuming the person wears the same outfit (easier, relies on color/texture).
  2. Cloth-Changing (CC): Recognizing someone even if they change their attire (harder, relies on gait, body shape, or accessories).

In the real world, surveillance systems face both scenarios simultaneously and must learn continuously from new data without forgetting what they previously learned—a challenge known as Lifelong Learning (LReID).

The authors identify a major gap: existing models usually treat SC and CC as separate problems. However, a feature that is highly discriminative for SC (like a bright red shirt) becomes "noise" in a CC scenario. The paper proposes the LReID-Hybrid task, requiring a model to handle both clothing states across an unending stream of diverse datasets.

Methods

The authors propose Contextualized Multimodal Lifelong Re-ID (CMLReID), which is built on the CLIP (Contrastive Language-Image Pre-training) architecture. It uses two primary novel modules:

1. Context-Aware Semantic Prompt (CASP) Learning

Instead of using static text descriptions (e.g., "a photo of a person"), CASP uses a dynamic context encoder. This encoder looks at the input image and estimates the "clothing state." * If the model detects a Same-Cloth scenario, it generates prompts that emphasize fine-grained clothing details. * If it detects a Cloth-Changing scenario, it shifts the prompt's focus toward invariant body attributes. This allows the model to "distill" visual cues into a unified textual space that is robust to outfit changes.

2. Adaptive Knowledge Fusion and Projection (AKFP) Strategy

To prevent the model from forgetting old tasks as it learns new ones, AKFP employs a Dual-Path Slow Learner.

  • Prototypes: The model maintains "prototypes" (generalized embeddings) for both SC and CC states. These are updated slowly using a momentum-based approach, acting as a stable memory of past knowledge.
  • Projection Loss: A new Clothing-State-Aware Projection Loss forces visual features to align with these stable prototypes. This ensures that even as the image encoder updates, it stays anchored to the fundamental characteristics of SC and CC identities.

Datasets

The study utilizes a comprehensive suite of ReID benchmarks to simulate a lifelong learning stream:

  • Seen Same-Cloth (SC): Market-1501 and MSMT17.
  • Seen Cloth-Changing (CC): LTCC and PRCC.
  • Unseen Generalization (SC): CUHK01, CUHK02, GRID, SenseReID, and PRID.
  • Unseen Generalization (CC): VC-Clothes and Celeb-ReID (light version).

The authors tested the model across six different training orders (e.g., starting with Market-1501 vs. starting with LTCC) to ensure the results weren't biased by the sequence of data.

Key Insights and Ideas

  • The Power of Multimodality: Textual descriptions are inherently more "abstract" than pixels. By forcing the model to align images with adaptive text prompts, the authors provide a "bridge" that helps the model ignore superficial clothing changes in favor of structural identity.
  • Context-Specific Feature Extraction: The "aha moment" here is that a single feature extraction strategy isn't enough. By explicitly classifying the clothing state first, the model can choose the "projection pathway" that makes the most sense for that specific image.
  • Stability via "Slow Learning": By updating prototypes with a very small momentum coefficient (0.001), the model accumulates knowledge across domains without the "spikes" in learning that typically lead to catastrophic forgetting.

Key Results

CMLReID consistently outperformed strong baselines like SFT and AKA across all metrics:

  • Overall Performance: On a sequence of four datasets, CMLReID achieved a Total Average mAP of 46.7% and Rank-1 accuracy of 60.5%, significantly higher than the SFT baseline (41.0% mAP / 57.4% R-1).
  • Cloth-Changing Robustness: In the difficult CC scenarios, the model reached 33.8% mAP, a notable jump from the 31.7% achieved by previous methods.
  • Order Independence: The model showed remarkable stability regardless of the order in which datasets were presented, with Total Average mAP fluctuating by less than 1% across six different sequences.
  • Human Validation: In a "top-5 retrieval" test, human annotators found the correct identity 75.3% of the time in CC scenarios using CMLReID, compared to 68.5% for the baseline.

Limitations and Future Work

  • Hyperparameter Sensitivity: The model relies on a weighting hyperparameter ($\lambda$) and a momentum coefficient ($\beta_s$). While the authors provide a sensitivity analysis, finding the "sweet spot" for these values in a completely open-ended real-world deployment might be challenging.
  • Computational Overhead: While the final evaluation only uses the image encoder, the training process involves alternating between prompt learning and feature projection, which may increase training time compared to simpler ReID methods.
  • Future Directions: The authors suggest that future work could explore even more granular "contexts" beyond just SC and CC, such as varying lighting conditions or extreme occlusions, to further harden the model for real-world surveillance.

Summary updated

Abstract

Person Re-Identification (ReID) has several challenges in real-world surveillance systems due to clothing changes (CCReID) and the need for maintaining continual learning (LReID). Previous existing methods either develop models specifically for one application, which is mostly a same-cloth (SC) setting or treat CCReID as its own separate sub-problem. In this work, we will introduce the LReID-Hybrid task with the goal of developing a model to achieve both SC and CC while learning in a continual setting. Mismatched representations and forgetting from one task to the next are significant issues, we address this with CMLReID, a CLIP-based framework composed of two novel tasks: (1) Context-Aware Semantic Prompt (CASP) that generates adaptive prompts, and also incorporates context to align richly multi-grained visual cues with semantic text space; and (2) Adaptive Knowledge Fusion and Projection (AKFP) which produces robust SC/CC prototypes through the use of a dual-path learner that aligns features with our Clothing-State-Aware Projection Loss. Experiments performed on a wide range of datasets and illustrate that CMLReID outperforms all state-of-the-art methods with strong robustness and generalization despite clothing variations and a sophisticated process of sequential learning.
BibTeX
@misc{long2025contextualized,
  author = {Robert Long and Rongxin Jiang and Mingrui Yan},
  title = {{Contextualized Multimodal Lifelong Person Re-Identification in Hybrid Clothing States}},
  year = {2025},
  eprint = {2509.11247},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.11247},
}
Figure