Abstract
Vision-Language Models (VLMs) like CLIP demonstrate exceptional zero-shot capabilities but often suffer performance degradation when encountering distribution shifts in real-world deployments. Test-Time Adaptation (TTA) offers a solution by adjusting models on-the-fly using unlabeled test data. However, existing cache-based TTA methods struggle with the "Entropy Trap"—overconfidence in incorrect predictions—and rigid decision boundaries. We introduce ReTA (Reliable Test-time Adaptation), a framework that employs Consistency-aware Entropy Reweighting (CER) to ensure cache purity and Diversity-driven Distribution Calibration (DDC) to model class embeddings as flexible Gaussian distributions. ReTA achieves state-of-the-art accuracy across diverse benchmarks while significantly outperforming competitors in computational efficiency.
Introduction
Large-scale VLMs are powerful tools for zero-shot classification, yet their performance typically declines when test data differs visually from training data (e.g., transitioning from natural photos to sketches or varying lighting conditions). To address this without labeled data, researchers utilize Test-Time Adaptation (TTA). A common strategy involves using a dynamic memory cache to store high-confidence samples for calibration.
Despite their popularity, cache-based methods face two primary reliability hurdles. First, they often rely on entropy (model confidence) to select samples, but VLMs are frequently overconfident about incorrect predictions under distribution shifts, leading to a "poisoned" cache. Second, VLMs typically use fixed text embeddings as classifiers, creating rigid decision boundaries that cannot accommodate the visual diversity of new domains.
Methodology
ReTA addresses these challenges through two primary modules designed to improve both sample selection and classifier flexibility.
Consistency-aware Entropy Reweighting (CER)
To avoid the entropy trap, CER implements a quality-control mechanism. Instead of relying on raw confidence, it generates a "voting committee" of diverse text prompts for each class. By projecting image features into the latent space and checking for prediction consistency across these prompts, CER identifies truly reliable samples. This consistency serves as a more accurate proxy for correctness than entropy alone, ensuring the cache remains populated with high-quality data.
Diversity-driven Distribution Calibration (DDC)
To overcome rigid boundaries, DDC shifts from treating classes as static points to modeling them as multivariate Gaussian distributions. Using the reliable samples vetted by CER, DDC updates the means of these distributions through lightweight residual learning. This allows the model's decision boundaries to dynamically stretch and adapt to the specific visual variations present in the test environment.
Experimental Results
ReTA was evaluated on two major benchmarks: a suite of 10 diverse image classification datasets for cross-dataset generalization and ImageNet along with four out-of-distribution (OOD) variants (ImageNet-A, -V2, -R, and -Sketch).
Accuracy and Robustness
ReTA achieved a state-of-the-art average accuracy of 71.37% on the cross-dataset benchmark, outperforming strong baselines such as DPE (69.40%) and BoostAdapter (68.68%). On ImageNet OOD variants, ReTA set a new benchmark with 66.07% average accuracy, demonstrating superior resilience against stylistic and domain shifts.
Cache Purity and Efficiency
A critical advantage of ReTA is its ability to maintain cache purity throughout the inference process. Unlike baseline methods that see cache accuracy degrade over time as errors accumulate, ReTA's CER module keeps the cache reliable. Furthermore, ReTA is highly efficient; by avoiding heavy encoder backpropagation and focusing on textual residuals, it runs approximately 1.6x faster than competing methods like BoostAdapter and DPE.
Conclusion
ReTA provides a robust and efficient framework for adapting VLMs to unseen visual domains. By replacing simple entropy metrics with consistency-based reweighting and evolving static embeddings into flexible distributions, ReTA overcomes the core limitations of previous TTA approaches. Future work may focus on optimizing the data augmentation strategies used for consistency checking to further reduce inference latency.
Summary updated