Abstract
Text-based Person Search (TBPS) aims to retrieve person images using natural language descriptions. Recently, Contrastive Language Image Pretraining (CLIP) has shown remarkable performance across various cross-modal tasks. This paper conducts a comprehensive empirical study of CLIP for TBPS, contributing a straightforward and strong baseline called TBPS-CLIP. By revisiting critical design considerations such as data augmentation and loss functions, the authors demonstrate that satisfactory performance can be achieved without sophisticated modules. Probing experiments on model generalization and compression further highlight the effectiveness of the proposed approach.
Introduction
Text-based Person Search (TBPS) is a fine-grained retrieval task where the goal is to identify a specific individual in an image database based on a natural language description, such as "a man in a red checkered shirt and khaki pants." While large-scale Vision-Language Pre-training (VLP) models like CLIP have revolutionized cross-modal tasks, TBPS research has often treated CLIP as a mere starting point, adding complex modules to adapt it to the domain. This study asks whether the potential of CLIP itself has been fully exploited and argues that a focus on training recipes over architectural complexity can yield superior results with significantly less computational overhead.
The TBPS-CLIP Framework
The authors propose TBPS-CLIP, a baseline that optimizes the standard CLIP architecture through four key dimensions: training tricks, data augmentation, loss functions, and model compression.
Methods and Training Recipes
- Training Tricks: Key improvements include global gradient back-propagation across GPUs, dropout for regularization, locking the initial layers of the image encoder, and utilizing "soft labels" to manage ambiguity in person descriptions.
- Data Augmentation: An Augmentation Pool strategy was introduced, randomly selecting two effective augmentations per image. Interestingly, RandomGrayscale was found to improve performance by forcing the model to learn robust features like texture and shape rather than relying solely on color.
- Loss Functions: The study explored several loss functions beyond standard contrastive loss, including Self-Supervision (SS), Multi-View Supervision (MVS), and Cyclic Image-Text Contrastive (C-ITC) loss to ensure geometric consistency in the feature space.
Empirical Insights and Probing Experiments
The study provides several critical insights into the behavior of CLIP for TBPS:
- Text Augmentation Sensitivity: While "Back Translation" improved diversity, "Random Swap" was detrimental. In TBPS, word order is crucial for spatial relationships; swapping words can destroy the semantic logic of a description.
- The "Middle-Layer Slump": Layer-wise probing revealed that the middle layers of the text encoder are largely redundant for TBPS. Freezing or dropping these layers results in minimal accuracy loss, offering a path for significant model compression.
Results and Efficiency
TBPS-CLIP achieves state-of-the-art performance across major benchmarks:
- CUHK-PEDES: Reached 65.05% Rank-1 accuracy, outperforming previous leaders like IRRA.
- Few-Shot Performance: In data-constrained scenarios (1% or 5% of training data), TBPS-CLIP significantly outperformed standard CLIP and adaptation methods like CoOp.
- Training Efficiency: TBPS-CLIP reaches peak performance in just 5 epochs (approx. 20 minutes), compared to 60 epochs (approx. 1.7 hours) for competing models like IRRA.
Conclusion
This empirical study demonstrates that a "back-to-basics" approach—optimizing simple data augmentations and loss functions—allows a standard CLIP model to outperform complex, state-of-the-art methods in TBPS. TBPS-CLIP serves as a lightweight, efficient, and strong baseline for the community. Future work may explore advanced prompt engineering or bridging the gap in zero-shot settings.
Summary updated