What Entropy-Based View Selection Cannot See: An Audit of Test-Time Adaptation for Vision-Language Models

Sivasuthan Sukumar, Uthayasanker ThayasivamarXiv 2026 · arXiv

This paper audits the popular test-time adaptation (TTA) pipeline for vision-language models, revealing that the widely-used entropy-based view selection is surprisingly ineffective on the errors that actually matter. By replacing model-derived confidence scores with a simple, "free" metadata signal—the crop area of augmented views—the authors achieve significant accuracy gains across 14 out of 15 benchmark datasets.

What Problem Does This Paper Address?

Vision-language models (VLMs) like CLIP are powerful zero-shot classifiers, but they struggle when test data differs from their training distribution (e.g., sketches vs. real photos). To fix this, researchers use Test-Time Adaptation (TTA). A standard TTA recipe involves creating multiple "views" of a test image (random crops and flips), scoring them to find the most "trustworthy" ones, and then averaging their predictions or updating the model.

The dominant way to score these views is prediction entropy: if the model is very certain about one class for a specific crop, that crop is trusted. This paper asks a fundamental question: Is entropy actually a good signal for identifying correct views? Does the model's own confidence help it recover from its own mistakes?

Methods

The authors conduct a rigorous "audit" by isolating each stage of the TTA pipeline:

  1. Stage Isolation: They measured the accuracy contribution of view selection, aggregation, and gradient updates independently.
  2. Controlled AUC Analysis: They used the Area Under the ROC Curve (AUC) to measure how well entropy ranks correct views above incorrect ones. Crucially, they focused on "reducible errors"—cases where the final averaged prediction is wrong, but at least one individual view was actually correct. They implemented a "split-half" control to ensure their findings weren't just statistical artifacts.
  3. Theoretical Analysis: They performed an exact mathematical derivation of the entropy objective's gradient to explain why it fails to enforce agreement between views.
  4. Crop Geometry Selection: As an alternative to entropy, they proposed ranking views by their crop area (the fraction of the original image the crop contains), a piece of metadata already generated during the augmentation process but previously ignored.

Datasets

The evaluation uses a standard 15-dataset benchmark:

  • ImageNet and Distribution Shifts: ImageNet, ImageNet-A (adversarial), ImageNet-K (corrupted), ImageNet-V2, and ImageNet-R (renditions).
  • Fine-grained Datasets: Flowers102, DTD (textures), OxfordPets, StanfordCars, UCF101 (actions), Caltech101, Food101, SUN397, Aircraft, and EuroSAT (satellite imagery).

Key Insights and Ideas

  • Selection is the Star, Gradient is the Extra: Almost all accuracy gains in TTA come from selecting the right views, not from the expensive gradient updates that many papers focus on. In fact, the gradient step's contribution was found to be statistically indistinguishable from zero.
Figure 1: The pipeline, and which of its stages produces the accuracy. (a) The augment, score, select, aggregate recipe TPT, ZERO and ADTE each declare. (b) Each stage measured with the other two held fixed, on ImageNet-A, in percentage points: the absolute gap between two accura- cies, so 40% to 59% is 19 of them. Selection produces almost all of the gain; the gradient step is indistinguishable from no update under a paired McNemar test [McNemar, 1947]. Concurrent work agrees by different means [Huang et al., 2026]; we claim the decomposition and its mechanism, not the direction.
  • The Entropy Collapse: While entropy seems to work well across all images, its performance "collapses" to near-chance levels on reducible errors. This means when the model is wrong and needs help choosing a better view, entropy is essentially useless at finding it.
Entropy's ranking performance collapses to chance on the errors that matter most.
  • Model "Class-Blindness": Standard scoring functions like entropy are "class-symmetric"—they only care about how concentrated a probability distribution is, not which class it favors. The authors show that preserving class identity is worth a massive 8.6 percentage point gain, but current models can't "see" this information in their own output distributions.
  • Metadata is a Hidden Gem: The augmentation procedure already knows how much of the image it's looking at. This "free" signal (crop area) is often more informative than the model's own complex confidence scores.

Key Results

  • Widespread Gains: Ranking views by crop area improved accuracy on 14 out of 15 benchmark datasets compared to standard entropy selection.
  • Significant Margin: On the suite average, crop geometry raised accuracy by 0.98 percentage points over Shannon entropy. Against the even stronger Tsallis entropy, it still won on 11 of 15 datasets.
  • Predictable Performance: The authors found they could predict which datasets would benefit from crop-area selection before running the method, simply by measuring the crop-area AUC on a small subset of data.

Limitations and Future Work

  • Lack of Composition: Interestingly, crop geometry gains don't simply add up with gains from better scoring functions (like Tsallis entropy). Once a stronger score is used, the benefit of crop area diminishes.
  • Unknown Mechanism: While the empirical relationship between crop area and accuracy is strong and predictive, the authors admit they don't yet have a definitive theoretical explanation for why it works so well.
  • Scope: The findings are specific to the "single-image" TTA regime. Batch-level adaptation or cache-based methods might behave differently.

Summary updated

Abstract

Vision-language models such as CLIP can classify images into categories they were never explicitly trained to recognize, but accuracy degrades when test images differ in style or source from their training data. A common remedy generates randomly cropped and flipped copies of each test image, called views, and combines their predictions. Since some views miss the object, methods must identify which views to trust. The dominant approach uses prediction entropy to measure confidence and retains the most certain views. Other methods use signals such as embedding typicality. Despite repeated refinements of these scoring functions, the information they rely on, the model's own response to each view, remains largely unexamined. We investigate this choice while holding other pipeline stages fixed. First, we show that view selection, rather than the gradient update, accounts for almost all accuracy gains. We then examine whether entropy identifies useful views when predictions are wrong, despite at least one view predicting the correct class. In these cases, entropy ranks correct views above incorrect ones no better than chance, recovering only about one-twentieth of the accuracy recovered by an oracle over the same views. This finding holds across two image encoders and under a control ruling out an obvious statistical artifact. An exact analysis explains this limitation: entropy measures confidence but not which class a view favors. We identify an unused signal already available in the augmentation pipeline: the fraction of the original image retained by each crop. Ranking views by this quantity improves accuracy on 14 of 15 benchmark datasets over entropy and on 11 of 15 over the strongest scoring function tested. A pre-method measurement also predicts which datasets benefit.
BibTeX
@misc{sukumar2026what,
  author = {Sivasuthan Sukumar and Uthayasanker Thayasivam},
  title = {{What Entropy-Based View Selection Cannot See: An Audit of Test-Time Adaptation for Vision-Language Models}},
  year = {2026},
  eprint = {2611.14686},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14686},
}
Figure