Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval

Shenshen Li, Chen He, Xing Xu, Fumin Shen, Yang Yang, Heng Tao ShenAAAI 2024 · DOI

TL;DR

This paper introduces an Adaptive Uncertainty-Based Learning (AUL) framework to improve text-based person retrieval. By explicitly modeling and filtering out unreliable matching pairs using Subjective Logic, and progressively refining multi-grained alignments, AUL achieves state-of-the-art performance across multiple benchmarks.

What Problem Does This Paper Address?

Text-based person retrieval aims to find a specific person in a collection of images using a textual description. The core challenge lies in bridging the "modality gap" between text and images, especially when dealing with significant variations within the same person (intra-class variation) and minimal differences between different people (inter-class variation). Existing methods often struggle with two main issues: 1) Matching ambiguity, caused by unreliable text-image pairs that are incorrectly matched due to superficial similarities. 2) One-sided cross-modal alignments, where models focus too much on one-to-one correspondences and fail to capture the inherent one-to-many relationships (e.g., a text description might cover general attributes, while an image contains many specific details). These issues lead to degraded retrieval performance and limited generalization.

Methods

The proposed Adaptive Uncertainty-Based Learning (AUL) framework tackles these problems with three key components:

  1. Uncertainty-aware Matching Filtration (UMF): This module uses Subjective Logic to quantify the "matching uncertainty" of text-image pairs. By assigning weights based on this uncertainty, it adaptively filters out unreliable pairs during training, ensuring that the model learns from high-confidence matches. This directly addresses the matching ambiguity problem.
  2. Uncertainty-based Alignment Refinement (UAR): This component aims to capture the one-to-many correspondence between text and images. It first constructs "uncertainty representations" for visual features by adding Gaussian noise. Then, it employs Alignment Progressive Learning (APL), which gradually shifts focus from coarse-grained (one-to-many) to fine-grained (one-to-one) alignments during training. This "easy-to-hard" approach helps the model build more comprehensive alignments.
  3. Cross-modal Masked Modeling (CMM): This module enhances the interaction between image and text modalities. It reconstructs masked image and text signals using information from the other modality. This component is further divided into:
  4. Cross-modal Masked Image Modeling (CMIM): Reconstructs masked image patches using text and unmasked image features.
  5. Cross-modal Masked Language Modeling (CMLM): Reconstructs masked text tokens using image and unmasked text features. This process helps mine deeper, fine-grained relationships between the two modalities.

The overall loss function combines losses from UMF, UAR, and CMM.

Datasets

The paper evaluates AUL on three standard benchmarks for text-based person retrieval:

  • CUHK-PEDES: Contains 40,206 images and 68,120 textual descriptions across 13,003 identities.
  • ICFG-PEDES: A large-scale dataset with 54,522 images, split into 34,674 training and 19,848 testing image-text pairs.
  • RSTPReid: Features 20,505 images from 4,101 identities, with each image having two textual descriptions.

Key Insights and Ideas

  • Subjective Logic for Matching Uncertainty: Applying Subjective Logic to quantify matching uncertainty is a novel way to identify and filter out unreliable text-image pairs, which are a major source of error in person retrieval.
  • Progressive Alignment is Crucial: The "easy-to-hard" approach of Alignment Progressive Learning (APL) in UAR, which starts with coarse-grained alignments and moves to fine-grained ones, is more effective than fixed or average weighting schemes. This reflects the natural learning process.
  • Cross-modal Masked Modeling for Deeper Relations: CMM, by forcing the model to reconstruct signals from one modality using the other, effectively bridges the modality gap and uncovers more nuanced relationships than standard attention mechanisms.
  • One-to-Many is Essential: The paper strongly emphasizes that capturing one-to-many correspondences is vital for robust text-based person retrieval, as text descriptions and image content rarely align perfectly one-to-one.

Key Results

AUL consistently achieves state-of-the-art performance across the evaluated datasets and settings (supervised, weakly supervised, and domain generalization). For instance, on RSTPReid, AUL achieves a remarkable 71.65% R@1, outperforming previous methods like RaSa and APTM by significant margins. Ablation studies confirm the effectiveness of each component: UMF significantly improves performance, UAR's progressive learning is beneficial, and CMM contributes substantially to the overall accuracy. Qualitative results show AUL retrieving more accurate and relevant images compared to APTM, demonstrating its ability to satisfy both coarse-grained and fine-grained retrieval requirements.

Limitations and Future Work

The paper acknowledges that while AUL significantly improves retrieval accuracy, there is still room for enhancement. The authors suggest exploring "other strategies to enhance the retrieval performance" in the future. Specific limitations are not explicitly detailed, but the ongoing pursuit of better performance implies that current methods, including AUL, can still be improved upon.

Summary updated

BibTeX
@inproceedings{li2024adaptive,
  author = {Shenshen Li and Chen He and Xing Xu and Fumin Shen and Yang Yang and Heng Tao Shen},
  title = {{Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28101},
}
Figure