Abstract
Text-based person re-identification (Re-ID) typically relies on large-scale datasets with identity-level annotations, which are costly and labor-intensive to obtain. This paper proposes a novel weakly supervised approach that utilizes only image-text pairs during training, removing the need for explicit identity labels. The core of the solution is the Cross-Modal Mutual Training (CMMT) framework, which incorporates Mutual Pseudo Label Refinement (MPLR) and a novel Text-IoU metric based on noun-phrase similarity. Experimental results on the CUHK-PEDES dataset demonstrate that CMMT achieves state-of-the-art performance, even surpassing several prominent fully supervised methods.
The Supervision Gap and Challenges
Text-based Person Re-ID involves retrieving images of a specific person from a large gallery based on a natural language description. Traditionally, training these models requires identity labels, where every image and text description is tagged with a specific person's ID. This process is difficult to scale across multiple non-overlapping camera views.
In the Weakly Supervised setting, the model only knows which text description belongs to which image (pairwise relationship) but lacks information on which pairs belong to the same person. This introduces two primary hurdles:
- Intra-class variation: The same individual can be described in diverse ways (textual variation) or captured from different angles and lighting conditions (visual variation).
- Cross-modal matching ambiguity: Without identity labels, it is challenging to determine if a description of "Person A" should match a different image of the same person.
The Cross-Modal Mutual Training (CMMT) Framework
To bridge the supervision gap, the authors propose the CMMT framework. This architecture leverages the strengths of both visual and textual modalities to provide mutual supervision.
The framework consists of several key components:
- Mutual Pseudo Label Refinement (MPLR): The model employs clustering to generate pseudo-labels for images and texts. Since clusters in one modality may be noisy, MPLR uses the known text-image pairs to refine the clusters in the other modality. For instance, if two images are in the same visual cluster, their corresponding text descriptions are likely related and should be grouped together.
- Text-IoU: This is a clever NLP-based heuristic. Using Noun Phrase Extraction (NPE), the model extracts key descriptors (e.g., "blue shirt," "black shoes") and calculates an "Intersection over Union" (IoU) score. High overlap suggests a likely match, providing a "soft label" to guide training.
- Text-IoU Guided Matching & Mining:
- CMPM Loss: Uses Text-IoU scores as a target distribution for matching, helping resolve which unpaired instances represent the same person.
- Hard Sample Mining (HSM): Identifies "hard negatives"—pairs that are close in embedding space but have low Text-IoU scores—forcing the model to learn more discriminative features.
Key Insights and Results
The most significant innovation is using NLP as a Supervision Proxy. By using noun-phrase overlap (Text-IoU) to replace human-provided identity labels, the model finds enough signal in natural language to converge effectively. Furthermore, the framework demonstrates that modalities can correct each other; visual clustering can fix errors in textual clustering and vice versa.
Performance on CUHK-PEDES
The researchers evaluated CMMT on the CUHK-PEDES dataset, which contains 11,003 identities and over 68,000 descriptions. Despite ignoring identity labels during training, CMMT achieved:
- Rank-1 Accuracy of 57.10%, setting a new record for weakly supervised Re-ID.
- Superiority over Supervised Models: Remarkably, CMMT outperformed several fully supervised models, such as ViTAA (55.97%), suggesting that the signals in natural language can be more robust than rigid identity labels.
- Ablation Success: Removing the MPLR or Text-IoU components led to significant performance drops, confirming their necessity.
Limitations and Future Work
While successful, the method relies heavily on the quality of Noun Phrase Extraction. Brief or abstract descriptions may result in a weak Text-IoU signal. Additionally, the study was limited to a single dataset; future research should explore noisier, "in-the-wild" data. Finally, the computational overhead of clustering and mutual refinement at every epoch remains a challenge for scaling to massive datasets.
Summary updated