Abstract: Grounded language-image pre-trained models have shown strong zero-shot generalization to various downstream object detection tasks. Despite their promising performance, the models rely heavily on the laborious prompt engineering. Existing works typically address this problem by tuning text prompts using downstream training data in a few-shot or fully supervised manner. However, a rarely studied problem is to optimize text prompts without using any annotations. In this paper, we delve into this problem and propose an Unsupervised Prompt Tuning framework for text-driven object detection, which is composed of two novel mean teaching mechanisms. In conventional mean teaching, the quality of pseudo boxes is expected to optimize better as the training goes on, but there is still a risk of overfitting noisy pseudo boxes. To mitigate this problem, 1) we propose Nested Mean Teaching, which adopts nested-annotation to supervise teacher-student mutual learning in a bi-level optimization manner; 2) we propose Dual Complementary Teaching, which employs an offline pre-trained teacher and an online mean teacher via data-augmentation-based complementary labeling so as to ensure learning without accumulating confirmation bias. By integrating these two mechanisms, the proposed unsupervised prompt tuning framework achieves significant performance improvement on extensive object detection datasets.
Introduction
Vision-language models (VLMs) like GLIP have revolutionized object detection by allowing users to detect objects using natural language prompts. While these models have impressive "zero-shot" capabilities, their performance often drops when applied to specific downstream tasks, such as autonomous driving in fog or detecting artistic sketches. Traditionally, researchers address this by tuning text prompts using labeled data. However, manually labeling bounding boxes is expensive and slow.
This paper introduces Unsupervised Prompt Tuning (UPT), a framework designed to optimize these prompts using only unlabeled images from the target domain. By shifting from supervised tuning to an unsupervised paradigm, UPT allows models to adapt to new environments without human intervention.
Methodology
The UPT framework builds on the "Mean Teacher" self-training paradigm, where a "Teacher" model generates pseudo-labels (predicted boxes) on unlabeled data to train a "Student" model. To handle the noise inherent in unsupervised detection, UPT introduces two major innovations:
1. Nested Mean Teaching (NMT)
Standard self-training risks overfitting to noisy, incorrect pseudo-labels. NMT introduces a bi-level optimization. An inner loop acts as a "practice round" where a ghosted teacher and student interact for $k$ steps to produce refined, "denoised" pseudo-labels. These high-quality labels are then used in the outer loop to perform the actual prompt update. This ensures that the student learns from increasingly accurate signals.
2. Dual Complementary Teaching (DCT)
To prevent the model from forgetting prior knowledge or falling into "confirmation bias" (repeatedly making the same mistakes), UPT employs two teachers:
- Offline Teacher: Uses original, fixed category names to ensure high-precision detections.
- Online Teacher: Uses evolving, learned prompts to improve recall of objects the original model might miss. By using different data augmentation strengths—weak for the stable Offline teacher and strong for the adaptive Online teacher—the model balances stability and plasticity.
Key Results
The authors evaluated UPT across 10 diverse datasets, including autonomous driving (Cityscapes, BDD100K), artistic styles (WaterColor), and specialized tasks (EgoHands).
- Performance Gains: On the Cityscapes dataset, UPT improved the baseline GLIP-T model from 33.8 mAP to 43.1 mAP (+9.3 points) without any human labels.
- Surpassing Supervised Baselines: Remarkably, UPT's unsupervised approach outperformed 5-shot supervised tuning on Cityscapes (43.1 vs 42.5 mAP), demonstrating that a large volume of unlabeled data can be more beneficial than a tiny handful of labeled examples.
- Domain Robustness: On the WaterColor dataset, UPT improved performance by 3.2 mAP over the zero-shot baseline, effectively bridging the gap between photographic pre-training and stylized tasks.
Discussion and Future Work
UPT demonstrates that tuning the language interface (prompts) is often more efficient and stable than retraining the entire visual backbone for domain adaptation. However, the framework faces a "cold start" problem: it requires the pre-trained model to have some initial zero-shot capability to generate the first set of pseudo-labels. Future work may explore applying unsupervised prompt tuning to even larger foundation models or extending it to video-based detection where temporal consistency can provide additional supervision.
Summary updated