Pretrain then Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

Jiahao Zhang, Yaxiong Wang, Zhedong ZhengSubmission to ICLR 2026 · Rejected

Abstract

Text-based person retrieval faces inherent limitations due to data scarcity, driven by stringent privacy constraints and the high cost of manual annotation. To mitigate this, existing methods usually rely on a \textbf{Pretrain-then-Finetune} paradigm, where models are first pretrained on synthetic person-caption data to establish cross-modal alignment, followed by fine-tuning on labeled real-world datasets. However, this paradigm lacks practicality in real-world deployment scenarios, where large-scale annotated target-domain data is typically inaccessible. In this work, we propose a new \textbf{Pretrain-then-Adapt} paradigm that eliminates reliance on extensive target-domain supervision. The key underpinning our approach is Uncertainty-Aware Test-Time Adaptation (UATTA), a framework enabling dynamic model adaptation using only unlabeled test data, with minimal computational overhead. UATTA introduces a bidirectional retrieval disagreement mechanism to estimate uncertainty, \ie, low uncertainty is assigned when an image-text pair ranks highly in both image-to-text and text-to-image retrieval, indicating high alignment; otherwise, high uncertainty is detected. This indicator drives test-time model recalibration without labels, effectively mitigating domain shift. We validate UATTA on four benchmarks, \ie, CUHK-PEDES, ICFG-PEDES, RSTPReid, and PAB, showing consistent improvements across both CLIP-based (one-stage) and XVLM-based (two-stage) frameworks. Ablation studies confirm that UATTA outperforms existing test-time adaptation strategies, establishing a new benchmark for label-efficient, deployable person retrieval systems.
BibTeX
@inproceedings{zhang2026pretrain,
  author = {Jiahao Zhang and Yaxiong Wang and Zhedong Zheng},
  title = {{Pretrain then Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2026},
  doi = {10.1145/3805712.3809598},
}
Figure