Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

Jiahao Zhang, Shaofei Huang, Yaxiong Wang, Zhedong Zheng

TL;DR

Text-based person search typically requires expensive, privacy-sensitive target domain labels to fine-tune models for real-world deployment. This paper introduces a "Pretrain-then-Adapt" paradigm using Uncertainty-Aware Test-Time Adaptation (UATTA) to dynamically optimize models using only unlabeled test data. By leveraging a novel bidirectional retrieval disagreement mechanism to filter out false positives, UATTA matches the performance of fully fine-tuned models while slashing post-training GPU time by over 99%.

What Problem Does This Paper Address?

Text-based person search—the task of finding a specific individual in an image gallery using a natural language description—is incredibly useful for applications like smart city management and locating missing persons. However, training these models is bottlenecked by data scarcity due to strict privacy regulations and the high cost of manual annotation.

To bypass this, standard approaches use a Pretrain-then-Finetune pipeline: models are pretrained on synthetic data and then fine-tuned on labeled real-world datasets. But what happens when you deploy a model to a new environment where you don't have any labeled target data?

Standard Test-Time Adaptation (TTA) methods try to adapt models on the fly using unlabeled test data by minimizing prediction entropy (making the model more confident). Unfortunately, under severe domain shifts, these models suffer from confirmation bias—they become highly overconfident in their own incorrect predictions (false positives), leading to catastrophic error accumulation.

Methods

The authors propose the Uncertainty-Aware Test-Time Adaptation (UATTA) framework, shifting the workflow to a Pretrain-then-Adapt paradigm. UATTA optimizes the model offline directly on the unlabeled test set through three core components:

  1. Cycle-Consistency Selection (CCS): Before adapting, the model filters out highly ambiguous queries. A text query is considered reliable only if it satisfies a round-trip retrieval constraint: the text retrieves a set of top-$K$ images, and those images in turn successfully retrieve the original text query within their top-$K$ matches.
  2. Bidirectional Retrieval Disagreement (BRD): To estimate uncertainty without ground-truth labels, the authors look at the symmetry of the model's predictions. If an image-text pair is truly a match, its text-to-image (T2I) retrieval probability should closely match its image-to-text (I2T) retrieval probability. A large disparity (disagreement) indicates high epistemic (model) uncertainty.
  3. Uncertainty-Weighted Entropy Minimization: UATTA updates the model's layer normalization parameters by minimizing entropy, but scales the gradients by the reciprocal of the BRD uncertainty score. This effectively suppresses gradients from overconfident false positives while amplifying updates from highly reliable, symmetrically aligned pairs.

The entire process is illustrated in the framework overview below:

Overview of the UATTA framework, showcasing Cycle-Consistency Selection and Bidirectional Retrieval Disagreement modules.

Datasets

The framework was validated across four challenging benchmarks:

  • CUHK-PEDES & ICFG-PEDES: Standard large-scale text-based person search datasets.
  • RSTPReid: A real-world dataset focusing on text-based person re-identification.
  • PAB (Person Anomaly Search): A benchmark specifically designed to test domain gaps, where the training data is entirely synthetic and the evaluation data consists of real-world images with human-corrected captions.

Key Insights and Ideas

  • Symmetry as a Proxy for Truth: Estimating true model uncertainty is mathematically intractable without labels. UATTA elegantly uses retrieval trajectory consistency as a computationally cheap, unsupervised proxy for uncertainty.
  • The False Positive Cluster: The authors empirically demonstrated that false positives heavily concentrate in high-disagreement regions, proving that bidirectional asymmetry is an excellent indicator of erroneous matches.
Figure 2: Statistical Overview of our Uncertainty Indicator on (a) RSTPReid [66] and (b) CUHK-PEDES [29]. We count the True Positives (TP) and False Positives (FP) sample number in the initial ranking list before adaptation according to the proposed uncertainty score. TP samples consistently clus- ter in the low-uncertainty region, while FP samples concen- trate in the high-uncertainty region across both benchmarks. Therefore, it could serve as the indicator for the test-time adaptation.
  • Lightweight Parameter Updates: Instead of updating the entire heavy Vision-Language Model (VLM), UATTA only tunes the affine parameters ($\gamma$ and $\beta$) of the final layer normalization blocks, preserving the robust pretrained feature manifold while adapting to the target domain.

Key Results

  • Massive Efficiency Gains: On the PAB benchmark, UATTA achieved a 99.6% reduction in post-train GPU time compared to full fine-tuning (0.08 hours on 1 GPU vs. 40.5 hours on 4 GPUs for an XVLM backbone), while delivering highly competitive performance (76.13% Rank@1 vs 81.95% for full fine-tuning).
Figure 1: Accuracy vs. efficiency trade-off on the PAB bench- mark [52]. Built upon a pretrained X-VLM [57] backbone, our method follows the Pretrain-then-Adapt paradigm and substantially improves retrieval performance with minimal adaptation cost (i.e., post-train GPU time). Compared with Pretrain-then-Finetune approaches, our method reduces post-train GPU time of adaptation by 99.6% while achiev- ing competitive performance, approaching or even matching strong Pretrain-then-Finetune methods such as IRRA [19]. We present Pretrain-then-Finetune results (i.e., CLIP [39] and IRRA [19]) as an upper bound on achievable performance under extensive finetuning. In contrast, our method reaches a favorable operating point close to this upper bound with significantly lower post-train time cost, highlighting its im- proved accuracy vs. efficiency trade-off under the Pretrain- then-Adapt setting.
  • Outperforming TTA Baselines: UATTA consistently outperformed existing state-of-the-art TTA methods (like Tent, SHOT, and TCR) across all datasets, achieving a +2.35% Rank@1 boost on RSTPReid over the unadapted baseline.

Limitations and Future Work

  • Strictness vs. Recall Trade-off: Under highly ambiguous text descriptions or low-quality images, both retrieval directions can become uniformly uncertain. In these edge cases, the Cycle-Consistency Selection module might discard hard but correct samples, which explains a slight drop observed in broader metrics like Rank@5 and Rank@10. The authors note that smoothly utilizing these "edge zone" samples is a key direction for future work.
  • Consistent-but-Wrong Errors: BRD measures self-consistency, not absolute correctness. If a pretrained model suffers from strong spurious correlations (e.g., consistently mapping a specific background to an incorrect text token in both directions), UATTA will not detect the error. Addressing these systematic calibration shifts remains an open challenge.

Summary updated

Abstract

Text-based person search faces inherent limitations due to data scarcity, driven by stringent privacy constraints and the high cost of manual annotation. To mitigate this, existing methods usually rely on a Pretrain-then-Finetune paradigm, where models are first pretrained on synthetic person-caption data to establish cross-modal alignment, followed by fine-tuning on labeled real-world datasets. However, this paradigm lacks practicality in real-world deployment scenarios, where large-scale annotated target-domain data is typically inaccessible. In this work, we propose a new Pretrain-then-Adapt paradigm that eliminates reliance on extensive target-domain supervision through an offline test-time adaptation manner, enabling dynamic model adaptation using only unlabeled test data with minimal post-train time cost. To mitigate overconfidence with false positives of previous entropy-based test-time adaptation, we propose an Uncertainty-Aware Test-Time Adaptation (UATTA) framework, which introduces a bidirectional retrieval disagreement mechanism to estimate uncertainty, i.e., low uncertainty is assigned when an image-text pair ranks highly in both image-to-text and text-to-image retrieval, indicating high alignment; otherwise, high uncertainty is detected. This indicator drives offline test-time model recalibration without labels, effectively mitigating domain shift. We validate UATTA on four benchmarks, i.e., CUHK-PEDES, ICFG-PEDES, RSTPReid, and PAB, showing consistent improvements across both CLIP-based (one-stage) and XVLM-based (two-stage) frameworks. Ablation studies confirm that UATTA outperforms existing offline test-time adaptation strategies, establishing a new benchmark for label-efficient, deployable person search systems. Our code is available at https://github.com/nkuzjh/UATTA.
BibTeX
@inproceedings{zhang2026pretrain,
  author = {Jiahao Zhang and Shaofei Huang and Yaxiong Wang and Zhedong Zheng},
  title = {{Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2026},
  doi = {10.1145/3805712.3809598},
}
Figure