An Empirical Study of CLIP for Text-based Person Search

Min Cao, Yang Bai, Ziyin Zeng, Mang Ye, Min Zhang

Abstract

Text-based Person Search (TBPS) aims to retrieve person images using natural language descriptions. Recently, Contrastive Language Image Pretraining (CLIP) has shown remarkable performance across various cross-modal tasks. This paper conducts a comprehensive empirical study of CLIP for TBPS, contributing a straightforward and strong baseline called TBPS-CLIP. By revisiting critical design considerations such as data augmentation and loss functions, the authors demonstrate that satisfactory performance can be achieved without sophisticated modules. Probing experiments on model generalization and compression further highlight the effectiveness of the proposed approach.

Introduction

Text-based Person Search (TBPS) is a fine-grained retrieval task where the goal is to identify a specific individual in an image database based on a natural language description, such as "a man in a red checkered shirt and khaki pants." While large-scale Vision-Language Pre-training (VLP) models like CLIP have revolutionized cross-modal tasks, TBPS research has often treated CLIP as a mere starting point, adding complex modules to adapt it to the domain. This study asks whether the potential of CLIP itself has been fully exploited and argues that a focus on training recipes over architectural complexity can yield superior results with significantly less computational overhead.

The TBPS-CLIP Framework

The authors propose TBPS-CLIP, a baseline that optimizes the standard CLIP architecture through four key dimensions: training tricks, data augmentation, loss functions, and model compression.

The TBPS-CLIP framework overview, illustrating the integration of data augmentation, training tricks, and various loss functions within the CLIP-based architecture.

Methods and Training Recipes

  • Training Tricks: Key improvements include global gradient back-propagation across GPUs, dropout for regularization, locking the initial layers of the image encoder, and utilizing "soft labels" to manage ambiguity in person descriptions.
  • Data Augmentation: An Augmentation Pool strategy was introduced, randomly selecting two effective augmentations per image. Interestingly, RandomGrayscale was found to improve performance by forcing the model to learn robust features like texture and shape rather than relying solely on color.
  • Loss Functions: The study explored several loss functions beyond standard contrastive loss, including Self-Supervision (SS), Multi-View Supervision (MVS), and Cyclic Image-Text Contrastive (C-ITC) loss to ensure geometric consistency in the feature space.

Empirical Insights and Probing Experiments

The study provides several critical insights into the behavior of CLIP for TBPS:

  • Text Augmentation Sensitivity: While "Back Translation" improved diversity, "Random Swap" was detrimental. In TBPS, word order is crucial for spatial relationships; swapping words can destroy the semantic logic of a description.
  • The "Middle-Layer Slump": Layer-wise probing revealed that the middle layers of the text encoder are largely redundant for TBPS. Freezing or dropping these layers results in minimal accuracy loss, offering a path for significant model compression.
Probing experiments on (a) TBPS-CLIP and (b) Simplified TBPS-CLIP, showing the Rank-1 accuracy when freezing or dropping layers in the text encoder.

Results and Efficiency

TBPS-CLIP achieves state-of-the-art performance across major benchmarks:

  • CUHK-PEDES: Reached 65.05% Rank-1 accuracy, outperforming previous leaders like IRRA.
  • Few-Shot Performance: In data-constrained scenarios (1% or 5% of training data), TBPS-CLIP significantly outperformed standard CLIP and adaptation methods like CoOp.
  • Training Efficiency: TBPS-CLIP reaches peak performance in just 5 epochs (approx. 20 minutes), compared to 60 epochs (approx. 1.7 hours) for competing models like IRRA.

Conclusion

This empirical study demonstrates that a "back-to-basics" approach—optimizing simple data augmentations and loss functions—allows a standard CLIP model to outperform complex, state-of-the-art methods in TBPS. TBPS-CLIP serves as a lightweight, efficient, and strong baseline for the community. Future work may explore advanced prompt engineering or bridging the gap in zero-shot settings.

Summary updated

Abstract

Text-based Person Search (TBPS) aims to retrieve the person images using natural language descriptions. Recently, Contrastive Language Image Pretraining (CLIP), a universal large cross-modal vision-language pre-training model, has remarkably performed over various cross-modal downstream tasks due to its powerful cross-modal semantic learning capacity. TPBS, as a fine-grained cross-modal retrieval task, is also facing the rise of research on the CLIP-based TBPS. In order to explore the potential of the visual-language pre-training model for downstream TBPS tasks, this paper makes the first attempt to conduct a comprehensive empirical study of CLIP for TBPS and thus contribute a straightforward, incremental, yet strong TBPS-CLIP baseline to the TBPS community. We revisit critical design considerations under CLIP, including data augmentation and loss function. The model, with the aforementioned designs and practical training tricks, can attain satisfactory performance without any sophisticated modules. Also, we conduct the probing experiments of TBPS-CLIP in model generalization and model compression, demonstrating the effectiveness of TBPS-CLIP from various aspects. This work is expected to provide empirical insights and highlight future CLIP-based TBPS research.
BibTeX
@inproceedings{cao2024empirical,
  author = {Min Cao and Yang Bai and Ziyin Zeng and Mang Ye and Min Zhang},
  title = {{An Empirical Study of CLIP for Text-based Person Search}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27801},
}
Figure