TL;DR
Test-Time Prompt Tuning (TPT) adapts vision-language models (VLMs) like CLIP to new domains on-the-fly, but its reliance on prediction entropy minimization severely inflates model overconfidence. This paper identifies a root cause—a massive modality gap concentrated in a single dominant embedding dimension—and proposes D-TPT, a lightweight method that maximizes entropy across feature dimensions to restore calibration and maintain competitive accuracy.
What Problem Does This Paper Address?
When deploying pretrained models like CLIP to unfamiliar test distributions, Test-time Prompt Tuning (TPT) optimizes learnable prompt tokens at inference time using unlabeled target data. Standard TPT does this by minimizing the entropy of model predictions across multiple augmented views. While effective at boosting accuracy, this pushes the model toward severe overconfidence and miscalibration (i.e., predicted probabilities become detached from true likelihoods of correctness).
Previous fixes (such as C-TPT and O-TPT) attempt to fix miscalibration by pushing different text embeddings apart (increasing inter-class text diversity or enforcing orthogonality). However, the authors find that higher text diversity does not consistently correlate with lower calibration error, suggesting that the underlying mechanism causing miscalibration in VLMs remained misunderstood.
Methods
Instead of spreading features out across different classes, the authors propose Dimensional Entropy Maximization (DEM), regularizing the distribution within individual class feature vectors:
- Identifying Dominant Dimensions: Contrastive VLMs suffer from a known "modality gap" where vision and language features inhabit distinct regions of the hypersphere. The authors show this gap is disproportionately driven by a single dominant feature dimension in text embeddings (TDD) and image embeddings (IDD).
-
Intra-Feature Uniformity via DEM: D-TPT introduces a regularizer $\mathcal{L}_{\text{DEM}}$ that penalizes deviation from uniformity across all dimensions for each normalized text embedding $\bar{t}_c$: $$\mathcal{L}_{\text{DEM}} = \frac{1}{C}\sum_{c=1}^C \text{KLD}(\sigma(\bar{t}_c) \parallel \mathcal{U})$$ where $\mathcal{U}$ is the discrete uniform distribution, $\sigma(\cdot)$ is softmax, and $\text{KLD}$ is Kullback–Leibler divergence.
-
Plug-and-Play Objective: The overall loss adds DEM directly to the standard TPT objective: $\mathcal{L}_{\text{D-TPT}} = \mathcal{L}_{\text{TPT}} + \lambda \mathcal{L}_{\text{DEM}}$. No architectural changes or extra model parameters are required.
Datasets
The authors evaluate D-TPT across standard VLM evaluation benchmarks:
- 11 Fine-Grained Classification Datasets: ImageNet, DTD, Flowers102, Food101, SUN397, FGVC-Aircraft, OxfordPets, Caltech101, UCF101, EuroSAT, and StanfordCars.
- 4 Out-of-Distribution / Natural Distribution Shift Datasets: ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-Sketch.
- Medical Imaging Domain (Appendix): Evaluated on COVID, RSNA18, and KatherColon using MedCLIP and PLIP backbones.
Key Insights and Ideas
- The Modality Gap Is Heavily Concentrated: The text-dominant dimension (index 362) and image-dominant dimension (index 442 in CLIP-ViT-B/16) appear consistently across all classes, images, and datasets.
- Dominant Dimensions Exert Disproportionate Sensitivity: Masking or perturbing these dominant dimensions causes huge swings in output probabilities. Artificially replacing them with class-wise mean values consistently decreases Expected Calibration Error (ECE) for both zero-shot CLIP and TPT.
- Geometric Mechanics on the Hypersphere:
- Standard TPT stretches text features along the tangent direction toward the image feature, drastically widening the logit range and causing overconfidence.
- By penalizing the spike in the dominant dimension, D-TPT forces energy to redistribute into binormal dimensions orthogonal to the tangent axis. This narrows the logit range, contracts the modality gap, and reins in inflated softmax confidences.
Key Results
- Fine-Grained Classification (CLIP-ViT-B/16): D-TPT achieves 64.72% accuracy and reduces ECE to 4.18%, outperforming baseline TPT (11.42% ECE), C-TPT (4.97% ECE), and O-TPT (4.78% ECE), even beating the unadapted zero-shot ECE (4.25%). It also reduces the Area Under the Risk–Coverage curve (AURC) to $181.67 \times 10^3$.
- Natural Distribution Shifts (CLIP-ViT-B/16): D-TPT drops average ECE on the four ImageNet shift variants to 3.83% (compared to 11.77% for TPT, 5.42% for C-TPT, and 5.14% for O-TPT).
- Broad Model and Domain Generalization: Similar calibration gains hold on ResNet backbones (CLIP-RN50) and transfer to medical prompt tuning under the BAPLe framework.
Limitations and Future Work
- Theoretical Gap Under Multi-View Augmentation: The geometric intuition is derived assuming a single image feature, whereas TPT in practice computes gradients over $N$ augmented image views.
- Lack of Closed-Form Theory: While the empirical connection between dominant dimensions, binormal dispersion, and logit ranges is clear, a fully rigorous mathematical theory connecting the modality gap to uncertainty calibration in multimodal contrastive spaces remains an open challenge.