Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations

Yiwen Liang, Hui Chen, Yizhe Xiong, Zihan Zhou, Mengyao Lyu, Zijia Lin, Shuaicheng Niu, Sicheng Zhao, Jungong Han, Guiguang Ding

Abstract

Vision-Language Models (VLMs) like CLIP demonstrate exceptional zero-shot capabilities but often suffer performance degradation when encountering distribution shifts in real-world deployments. Test-Time Adaptation (TTA) offers a solution by adjusting models on-the-fly using unlabeled test data. However, existing cache-based TTA methods struggle with the "Entropy Trap"—overconfidence in incorrect predictions—and rigid decision boundaries. We introduce ReTA (Reliable Test-time Adaptation), a framework that employs Consistency-aware Entropy Reweighting (CER) to ensure cache purity and Diversity-driven Distribution Calibration (DDC) to model class embeddings as flexible Gaussian distributions. ReTA achieves state-of-the-art accuracy across diverse benchmarks while significantly outperforming competitors in computational efficiency.

Introduction

Large-scale VLMs are powerful tools for zero-shot classification, yet their performance typically declines when test data differs visually from training data (e.g., transitioning from natural photos to sketches or varying lighting conditions). To address this without labeled data, researchers utilize Test-Time Adaptation (TTA). A common strategy involves using a dynamic memory cache to store high-confidence samples for calibration.

Despite their popularity, cache-based methods face two primary reliability hurdles. First, they often rely on entropy (model confidence) to select samples, but VLMs are frequently overconfident about incorrect predictions under distribution shifts, leading to a "poisoned" cache. Second, VLMs typically use fixed text embeddings as classifiers, creating rigid decision boundaries that cannot accommodate the visual diversity of new domains.

Methodology

ReTA addresses these challenges through two primary modules designed to improve both sample selection and classifier flexibility.

Consistency-aware Entropy Reweighting (CER)

To avoid the entropy trap, CER implements a quality-control mechanism. Instead of relying on raw confidence, it generates a "voting committee" of diverse text prompts for each class. By projecting image features into the latent space and checking for prediction consistency across these prompts, CER identifies truly reliable samples. This consistency serves as a more accurate proxy for correctness than entropy alone, ensuring the cache remains populated with high-quality data.

Diversity-driven Distribution Calibration (DDC)

To overcome rigid boundaries, DDC shifts from treating classes as static points to modeling them as multivariate Gaussian distributions. Using the reliable samples vetted by CER, DDC updates the means of these distributions through lightweight residual learning. This allows the model's decision boundaries to dynamically stretch and adapt to the specific visual variations present in the test environment.

Comparison of rigid decision boundaries (left) versus ReTA's Diversity-driven Distribution Calibration (right), which models classes as Gaussian distributions to handle visual shifts.

Experimental Results

ReTA was evaluated on two major benchmarks: a suite of 10 diverse image classification datasets for cross-dataset generalization and ImageNet along with four out-of-distribution (OOD) variants (ImageNet-A, -V2, -R, and -Sketch).

Accuracy and Robustness

ReTA achieved a state-of-the-art average accuracy of 71.37% on the cross-dataset benchmark, outperforming strong baselines such as DPE (69.40%) and BoostAdapter (68.68%). On ImageNet OOD variants, ReTA set a new benchmark with 66.07% average accuracy, demonstrating superior resilience against stylistic and domain shifts.

Cache Purity and Efficiency

A critical advantage of ReTA is its ability to maintain cache purity throughout the inference process. Unlike baseline methods that see cache accuracy degrade over time as errors accumulate, ReTA's CER module keeps the cache reliable. Furthermore, ReTA is highly efficient; by avoiding heavy encoder backpropagation and focusing on textual residuals, it runs approximately 1.6x faster than competing methods like BoostAdapter and DPE.

Performance analysis showing (a) cache accuracy and (b) overall accuracy across testing progress, highlighting ReTA's ability to maintain high cache purity compared to baselines.

Conclusion

ReTA provides a robust and efficient framework for adapting VLMs to unseen visual domains. By replacing simple entropy metrics with consistency-based reweighting and evolving static embeddings into flexible distributions, ReTA overcomes the core limitations of previous TTA approaches. Future work may focus on optimizing the data augmentation strategies used for consistency checking to further reduce inference latency.

Summary updated

BibTeX
@inproceedings{liang2025advancing,
  author = {Yiwen Liang and Hui Chen and Yizhe Xiong and Zihan Zhou and Mengyao Lyu and Zijia Lin and Shuaicheng Niu and Sicheng Zhao and Jungong Han and Guiguang Ding},
  title = {{Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755607},
}

Loading related content...

Figure