O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models

Ashshak Sharifdeen, Muhammad Akhtar Munir, Sanoojan Baliah, Salman Khan, Muhammad Haris Khan

Abstract

Test-time prompt tuning for vision-language models (VLMs) is getting attention because of their ability to learn with unlabeled data without fine-tuning. Although test-time prompt tuning methods for VLMs can boost accuracy, the resulting models tend to demonstrate poor calibration, which casts doubts on the reliability and trustworthiness of these models. Notably, more attention needs to be devoted to calibrating the test-time prompt tuning in vision-language models. To this end, we propose a new approach, called O-TPT that introduces orthogonality constraints on the textual features corresponding to the learnable prompts for calibrating test-time prompt tuning in VLMs. Towards introducing orthogonality constraints, we make the following contributions. First, we uncover new insights behind the suboptimal calibration performance of existing methods relying on textual feature dispersion. Second, we show that imposing a simple orthogonalization of textual features is a more effective approach towards obtaining textual dispersion. We conduct extensive experiments on various datasets with different backbones and baselines. The results indicate that our method consistently outperforms the prior state of the art in significantly reducing the overall average calibration error. Also, our method surpasses the zero-shot calibration performance on fine-grained classification tasks.

Introduction

Vision-Language Models (VLMs) like CLIP have revolutionized computer vision with their zero-shot capabilities. To further adapt these models to specific, unseen datasets during inference, Test-Time Prompt Tuning (TPT) has emerged as a popular technique. TPT optimizes text prompts (e.g., "a photo of a [class]") using unlabeled data at test time, significantly boosting accuracy.

However, this accuracy gain often comes at the cost of poor calibration. Models become dangerously overconfident, assigning high probabilities to incorrect guesses. In high-stakes domains like healthcare or autonomous driving, such overconfidence is a major liability. While previous methods like C-TPT attempted to "disperse" text features using L2 distance, research shows that simply pushing features apart is insufficient if they still point in similar directions in the embedding space.

Methodology: Enforcing Orthogonality

The proposed O-TPT (Orthogonality-constrained Test-time Prompt Tuning) moves beyond simple dispersion by enforcing orthogonality among textual features. In VLMs, text features are typically normalized and reside on a unit hypersphere. O-TPT introduces a regularization term to the TPT loss function:

$$\lambda \|EE^T - I_C\|_2^2$$

Where $E$ is the matrix of text features and $I_C$ is the identity matrix. This constraint pushes the pairwise cosine similarity of all class features toward zero, forcing the model to ensure that the "direction" for one class is as distinct as possible from another. To stabilize this optimization during single-step updates, the method also employs a Householder transformation to efficiently navigate the feature space.

Conceptual comparison of Angular optimization (O-TPT) versus distance-based ATFD optimization (C-TPT) on the hypersphere.

Experimental Results

O-TPT was evaluated across a massive battery of benchmarks, including general classification (ImageNet), fine-grained datasets (DTD, OxfordPets, StanfordCars), and medical imaging (COVID-19 detection, skin lesions).

Across 11 datasets using a ViT-B/16 backbone, O-TPT reduced the average Expected Calibration Error (ECE) to 4.23, compared to 5.13 for C-TPT and 11.6 for standard TPT. Remarkably, O-TPT often achieved better calibration than the original zero-shot CLIP model while simultaneously providing the accuracy boost of prompt tuning.

Radar chart showing ECE across various datasets; O-TPT (Ours) consistently achieves the lowest error (innermost polygon) compared to baselines.

In specialized tasks like COVID-19 detection, O-TPT drastically reduced ECE from 15.64 to 5.97, making the models significantly more viable for clinical decision support.

Key Insights and Analysis

  • Angle Over Distance: The core discovery is that calibration error is strongly correlated with the angular separation (cosine similarity) of text features. If features are too close in angle, the model struggles to define clear boundaries, leading to overconfidence.
  • Hypersphere Utilization: Existing L2-based methods often fail in challenging cases where features remain clustered. By enforcing orthogonality, O-TPT forces the model to utilize the full geometry of the hypersphere.
  • Balanced Reliability: Reliability diagrams indicate that O-TPT doesn't just fix overconfidence; it also rectifies underconfidence in certain datasets, creating a more balanced and trustworthy predictive distribution.

Conclusion

O-TPT demonstrates that orthogonality is a powerful constraint for maintaining model reliability during test-time adaptation. By focusing on the angular relationship between class features, O-TPT ensures that VLMs remain "honest" about their predictions. Future work will explore hyperparameter sensitivity across different distribution shifts and the scalability of these constraints for larger multi-modal variants.

Summary updated

Abstract

Test-time prompt tuning for vision-language models (VLMs) is getting attention because of their ability to learn with unlabeled data without fine-tuning. Although test-time prompt tuning methods for VLMs can boost accuracy, the resulting models tend to demonstrate poor calibration, which casts doubts on the reliability and trustworthiness of these models. Notably, more attention needs to be devoted to calibrating the test-time prompt tuning in vision-language models. To this end, we propose a new approach, called O-TPT that introduces orthogonality constraints on the textual features corresponding to the learnable prompts for calibrating test-time prompt tuning in VLMs. Towards introducing orthogonality constraints, we make the following contributions. First, we uncover new insights behind the suboptimal calibration performance of existing methods relying on textual feature dispersion. Second, we show that imposing a simple orthogonalization of textual features is a more effective approach towards obtaining textual dispersion. We conduct extensive experiments on various datasets with different backbones and baselines. The results indicate that our method consistently outperforms the prior state of the art in significantly reducing the overall average calibration error. Also, our method surpasses the zero-shot calibration performance on fine-grained classification tasks.
BibTeX
@inproceedings{sharifdeen2025o,
  author = {Ashshak Sharifdeen and Muhammad Akhtar Munir and Sanoojan Baliah and Salman Khan and Muhammad Haris Khan},
  title = {{O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=zRfmx3O2qc},
}
Figure