Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation

Taehoon Kim, Wonjun Hwang, Hyunsouk ChoarXiv 2026 · arXiv

This paper introduces CDIM (Cross-modal DINO-anchored Information Maximization), a robust framework for adapting vision-language models like CLIP to new, corrupted data environments in real-time. By ensembling CLIP with a frozen, self-supervised DINO v2 backbone and employing a gated information-maximization objective, it successfully breaks the "confirmation bias" loop that typically causes these models to fail under distribution shifts.

TL;DR

CDIM fixes "modality bias" in CLIP-based test-time adaptation by ensembling it with a frozen DINO v2 backbone and using a gated information-maximization objective. This prevents the model from making confident, repeated mistakes when faced with corrupted images, significantly improving both accuracy and calibration.

What Problem Does This Paper Address?

Vision-language models (VLMs) like CLIP are powerful because they can classify images using arbitrary text labels (zero-shot). However, they have a hidden weakness: modality bias. When an image is corrupted (e.g., by noise, blur, or weather), CLIP’s visual encoder often projects these distorted features onto incorrect text labels with extremely high confidence.

In a standard Test-Time Adaptation (TTA) scenario—where a model tries to improve itself on unlabeled data during inference—this high confidence is a disaster. Common techniques like entropy minimization assume the model's high-confidence predictions are likely correct and "sharpen" them. Under corruption, this creates a confirmation-bias loop: the model makes a confident mistake, trains on that mistake, and becomes even more certain of its error, eventually causing the predicted label distribution to collapse.

Methods

The authors propose CDIM, which tackles modality bias at three distinct levels of the adaptation pipeline:

  1. Representation Level (DINO-anchored Ensemble): They pair CLIP with a frozen self-supervised backbone (DINO v2). Unlike CLIP, DINO is trained only on images and is more sensitive to low-level visual structures. By projecting CLIP’s text prototypes into DINO’s feature space using a cross-covariance logit ensemble, they create a "visual anchor" that is less susceptible to CLIP’s text-driven biases.
  2. Optimization Level (Gated Information Maximization): Instead of simple entropy minimization, they use Gated Test-time Information Maximization (GT-IM). This objective balances per-sample sharpening with batch-level diversity (preventing the model from predicting the same class for everything). Crucially, an adaptive confidence gate filters out the most unreliable (noisiest) samples from the gradient updates.
  3. Prediction Level (Online Logit Adjustment): Even after representation and optimization fixes, some bias might remain. CDIM uses Online Logit Adjustment (OLA), which tracks predicted class frequencies and applies a penalty to over-predicted classes, ensuring a more balanced and realistic output distribution.
Comparison of CDIM with standard TTA paradigms and its impact on model reliability.

Datasets

The authors evaluated CDIM on several standard benchmarks for model robustness:

  • ImageNet-C, CIFAR10-C, and CIFAR100-C: The primary benchmarks, containing 15 types of common corruptions (noise, blur, weather, digital) at various severities.
  • Domain Generalization Benchmarks: VisDA-C, OfficeHome, PACS, and VLCS.
  • Natural Distribution Shifts: ImageNet-A (adversarial), ImageNet-V2, ImageNet-R (renditions), and ImageNet-S (sketch).

Key Insights and Ideas

  • DINO as a "Text-Free" Anchor: The core "aha" moment is realizing that CLIP’s failure is tied to its language supervision. By bringing in a model (DINO) that has never seen a word, you provide a necessary reality check for the visual features.
  • Breaking the Confirmation Loop: Standard TTA is like an echo chamber. CDIM breaks this by shifting the confidence of incorrect predictions toward more moderate values. As shown in Figure 1, CDIM prevents the "near-perfect confidence" spikes for wrong answers that plague standard methods.
  • Quantile-Based Gating: Rather than using a fixed threshold to reject noisy samples (which might reject an entire "hard" batch), CDIM uses a quantile-based gate. This ensures the model always learns from the relatively best samples in every batch, maintaining a stable adaptation signal.
Figure 1: Confidence of incorrect predictions on ImageNet-C (all corruptions, severity 5). CLIP-based entropy minimization (red) assigns high confidence even to incorrect predictions under corruption. CDIM (blue) shifts the confidence of incorrect predictions toward moderate values, which yields a flatter dis- tribution. The corresponding reliability diagram and ECE with 10 equal-width bins are shown in Figure 2b.

Key Results

CDIM sets a new state-of-the-art for CLIP-based online test-time adaptation:

  • Superior Accuracy: It outperforms the strongest previous baseline (SubTTA) by +4.8% on ImageNet-C, +3.5% on CIFAR10-C, and +3.7% on CIFAR100-C.
  • Better Calibration: The model is much more "honest" about what it doesn't know. Its Expected Calibration Error (ECE) on ImageNet-C drops from 0.400 (CLIP+TENT) to just 0.048.
  • Robustness to Natural Shifts: Beyond synthetic corruptions, CDIM remains competitive on natural distribution shifts like ImageNet-R and ImageNet-Sketch, showing that its benefits generalize to real-world "in the wild" scenarios.
Table 1: Online TTA accuracy (%) with ViT-B/16 at severity 5. Top-1 and Top-2 are shown in bold and underline, respectively.

Limitations and Future Work

  • Uniform Prior Assumption: Both the marginal entropy term and the logit adjustment assume a roughly uniform distribution of classes in the test stream. Performance might degrade on highly imbalanced or temporally correlated data.
  • Computational Overhead: Adding a second backbone (DINO v2) requires an additional forward pass per batch, increasing latency and memory usage compared to single-backbone methods.
  • Fixed Hyperparameters: The discard ratio $\rho$ is currently fixed across all benchmarks. The authors suggest that a severity-adaptive ratio could further improve results.
  • Scope: The current work focuses on image classification; extending this cross-modal anchoring to object detection or segmentation is a promising future direction.

Summary updated

Abstract

Vision-language models such as CLIP enable zero-shot classification from arbitrary class names, but under test-time corruptions, their visual encoder projects distorted features onto incorrect textual anchors with high confidence: a phenomenon we term modality bias. Standard entropy minimization reinforces these overconfident errors, which creates a confirmation-bias loop and collapses the predicted label distribution. We propose an online test-time adaptation (TTA) method that breaks this loop at three levels. At the representation level, a frozen self-supervised backbone (e.g., DINO v2) is combined with CLIP through a cross-covariance logit ensemble. CLIP text prototypes are projected into the feature space of this text-free backbone, so the self-supervised branch classifies native image features and is less affected by the text bias of CLIP. At the optimization level, a gated test-time information-maximization objective pairs per-sample sharpening with batch-level diversity, while an adaptive gate excludes unreliable samples from the gradient. At the prediction level, an online logit adjustment corrects the residual class-distribution skew. CDIM outperforms the compared CLIP-based online TTA baselines on ImageNet-C, CIFAR10-C, and CIFAR100-C, and remains competitive in continual and mixed-domain TTA. Extensive analyses and ablation studies verify the contribution of each of its components. Our code is available at https://github.com/B1ueC1oud/CDIM
BibTeX
@misc{kim2026bridging,
  author = {Taehoon Kim and Wonjun Hwang and Hyunsouk Cho},
  title = {{Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation}},
  year = {2026},
  eprint = {2611.04489},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04489},
}
Figure