Enhancing Cross-modal Completion and Alignment for Unsupervised Incomplete Text-to-Image Person Retrieval

Tiantian Gong, Junsheng Wang, Liyan ZhangIJCAI 2024

Introduction

Traditional text-to-image person retrieval (a sub-field of Person Re-Identification or ReID) typically relies on two major assumptions: full supervision, where every image-text pair is labeled with a specific identity, and complete modality, where every training sample consists of both an image and its corresponding text description.

In real-world applications, these assumptions are often violated. Labeling large datasets is cost-prohibitive, and data collection frequently results in "incomplete" sets where images lack descriptions or vice versa. This paper introduces ECCA (Enhancing Cross-modal Completion and Alignment), a framework designed to navigate these "messy" data scenarios without relying on identity labels.

Problem Statement: Incomplete Data and Ambiguity

Comparison of supervised vs. unsupervised settings and data issues. Source, p. 1

Comparison of supervised vs. unsupervised settings and data issues.

As illustrated in Figure 3, moving from a fully supervised to an unsupervised modal setting introduces significant challenges. Beyond the lack of labels, models must contend with Incomplete Data (missing images or text) and Matching Ambiguity, where it is unclear which disparate descriptions belong to the same individual. Previous models struggle to align "orphaned" data points or resolve these ambiguities without ground-truth guidance.

The ECCA Framework

The authors propose a four-part methodology to reconstruct missing information and align features across modalities:

  1. High Semantic Similarity Neighbor Generation (HSNG): To address missing data, ECCA identifies $k$-reciprocal nearest neighbors across modalities using a Jaccard distance metric. This ensures that neighbors are mutually similar, reducing the risk of using "false friends" for reconstruction.
  2. Feature-level Missing Modality Completion (FMC): Instead of generating raw pixels or text, ECCA reconstructs missing features in the embedding space. It builds a relational graph and weights the features of semantically similar neighbors to fill in the gaps.
  3. Weighted Inter-instance Granularity Alignment (WIGA): To resolve matching ambiguity, ECCA extracts noun phrases from text descriptions using NLTK and calculates the Intersection over Union (IoU). This overlap serves as a "matching probability," allowing the model to align similar instances even without explicit identity labels.
  4. Enhanced Prototype-wise Granularity Alignment (EPGA): The model maps both images and texts to a set of "shared prototypes"—representative vectors for semantic concepts like "backpack" or "blue jeans"—ensuring cross-modal alignment at a conceptual level.

Experimental Evaluation

ECCA was evaluated on the CUHK-PEDES and ICFG-PEDES datasets under varying levels of data incompleteness (Easy, Medium, and Hard).

In the "Hard" setting for CUHK-PEDES—where only 10% of data pairs are complete—ECCA achieved a 56.38% Rank-1 accuracy. This represents a nearly 6% improvement over the previous state-of-the-art unsupervised method (IRRA). Notably, ECCA's performance remained stable even as the data became increasingly incomplete, whereas competing models saw significant degradation.

Conclusion and Future Work

ECCA demonstrates that feature-level completion and noun-phrase-based alignment are powerful tools for unsupervised person retrieval. By prioritizing reliability in neighbor selection and leveraging text as a proxy for identity, ECCA effectively handles the noise and gaps inherent in real-world data. Future research may explore integrating Large Language Models (LLMs) to further enhance semantic understanding and alignment.

Summary updated

BibTeX
@inproceedings{gong2024enhancing,
  author = {Tiantian Gong and Junsheng Wang and Liyan Zhang},
  title = {{Enhancing Cross-modal Completion and Alignment for Unsupervised Incomplete Text-to-Image Person Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure