Weakly Supervised Text-based Person Re-Identification

Shizhen Zhao, Changxin Gao, Yuanjie Shao, Wei-Shi Zheng, Nong Sang

Abstract

Text-based person re-identification (Re-ID) typically relies on large-scale datasets with identity-level annotations, which are costly and labor-intensive to obtain. This paper proposes a novel weakly supervised approach that utilizes only image-text pairs during training, removing the need for explicit identity labels. The core of the solution is the Cross-Modal Mutual Training (CMMT) framework, which incorporates Mutual Pseudo Label Refinement (MPLR) and a novel Text-IoU metric based on noun-phrase similarity. Experimental results on the CUHK-PEDES dataset demonstrate that CMMT achieves state-of-the-art performance, even surpassing several prominent fully supervised methods.

The Supervision Gap and Challenges

Text-based Person Re-ID involves retrieving images of a specific person from a large gallery based on a natural language description. Traditionally, training these models requires identity labels, where every image and text description is tagged with a specific person's ID. This process is difficult to scale across multiple non-overlapping camera views.

Comparison of supervision paradigms and inherent challenges in text-based person Re-ID.

In the Weakly Supervised setting, the model only knows which text description belongs to which image (pairwise relationship) but lacks information on which pairs belong to the same person. This introduces two primary hurdles:

  1. Intra-class variation: The same individual can be described in diverse ways (textual variation) or captured from different angles and lighting conditions (visual variation).
  2. Cross-modal matching ambiguity: Without identity labels, it is challenging to determine if a description of "Person A" should match a different image of the same person.

The Cross-Modal Mutual Training (CMMT) Framework

To bridge the supervision gap, the authors propose the CMMT framework. This architecture leverages the strengths of both visual and textual modalities to provide mutual supervision.

Overview of the CMMT framework architecture, featuring dual encoders, memory banks, and the MPLR module.

The framework consists of several key components:

  • Mutual Pseudo Label Refinement (MPLR): The model employs clustering to generate pseudo-labels for images and texts. Since clusters in one modality may be noisy, MPLR uses the known text-image pairs to refine the clusters in the other modality. For instance, if two images are in the same visual cluster, their corresponding text descriptions are likely related and should be grouped together.
  • Text-IoU: This is a clever NLP-based heuristic. Using Noun Phrase Extraction (NPE), the model extracts key descriptors (e.g., "blue shirt," "black shoes") and calculates an "Intersection over Union" (IoU) score. High overlap suggests a likely match, providing a "soft label" to guide training.
  • Text-IoU Guided Matching & Mining:
    • CMPM Loss: Uses Text-IoU scores as a target distribution for matching, helping resolve which unpaired instances represent the same person.
  • Hard Sample Mining (HSM): Identifies "hard negatives"—pairs that are close in embedding space but have low Text-IoU scores—forcing the model to learn more discriminative features.

Key Insights and Results

The most significant innovation is using NLP as a Supervision Proxy. By using noun-phrase overlap (Text-IoU) to replace human-provided identity labels, the model finds enough signal in natural language to converge effectively. Furthermore, the framework demonstrates that modalities can correct each other; visual clustering can fix errors in textual clustering and vice versa.

Performance on CUHK-PEDES

The researchers evaluated CMMT on the CUHK-PEDES dataset, which contains 11,003 identities and over 68,000 descriptions. Despite ignoring identity labels during training, CMMT achieved:

  • Rank-1 Accuracy of 57.10%, setting a new record for weakly supervised Re-ID.
  • Superiority over Supervised Models: Remarkably, CMMT outperformed several fully supervised models, such as ViTAA (55.97%), suggesting that the signals in natural language can be more robust than rigid identity labels.
  • Ablation Success: Removing the MPLR or Text-IoU components led to significant performance drops, confirming their necessity.

Limitations and Future Work

While successful, the method relies heavily on the quality of Noun Phrase Extraction. Brief or abstract descriptions may result in a weak Text-IoU signal. Additionally, the study was limited to a single dataset; future research should explore noisier, "in-the-wild" data. Finally, the computational overhead of clustering and mutual refinement at every epoch remains a challenge for scaling to massive datasets.

Summary updated

Abstract

The conventional text-based person re-identification methods heavily rely on identity annotations. However, this labeling process is costly and time-consuming. In this paper, we consider a more practical setting called weakly supervised text-based person re-identification, where only the text-image pairs are available without the requirement of annotating identities during the training phase. To this end, we propose a Cross-Modal Mutual Training (CMMT) framework. Specifically, to alleviate the intra-class variations, a clustering method is utilized to generate pseudo labels for both visual and textual instances. To further re-fine the clustering results, CMMT provides a Mutual Pseudo Label Refinement module, which leverages the clustering results in one modality to refine that in the other modality constrained by the text-image pairwise relationship. Mean-while, CMMT introduces a Text-IoU Guided Cross-Modal Projection Matching loss to resolve the cross-modal matching ambiguity problem. A Text-IoU Guided Hard Sample Mining method is also proposed for learning discriminative textual-visual joint embeddings. We conduct extensive experiments to demonstrate the effectiveness of the proposed CMMT, and the results show that CMMT performs favorably against existing text-based person re-identification methods. Our code will be available at https://github.com/X-BrainLab/WS_Text-ReID.
BibTeX
@inproceedings{zhao2021weakly,
  author = {Shizhen Zhao and Changxin Gao and Yuanjie Shao and Wei-Shi Zheng and Nong Sang},
  title = {{Weakly Supervised Text-based Person Re-Identification}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01120},
}
Figure