Hierarchical Gap Priors for Test-Time Adaptation of CLIP

Vision-language models such as CLIP classify an image by comparing its embedding with the embeddings of text prompts like "a photo of a forest." A well-documented quirk makes this comparison awkward. Image embeddings and text embeddings do not mix in the shared space. They sit in two separate clusters a roughly constant distance apart, which the literature calls the modality gap. Methods that adapt CLIP on the fly to a new, unlabeled stream of test images often treat each class's text embedding as a prior guess for where that class's images will fall, then update the guess as images arrive. Because of the gap, that prior guess is wrong in a systematic and predictable way. We propose to model this error explicitly. A class's image center equals its text embedding, plus one displacement shared by all classes (the gap), plus a small class-specific residual. Each term gets a Gaussian prior whose mean and covariance are measured once, offline, from matched image-caption pairs.

The structure helps for a specific reason. The shared displacement can be estimated from every incoming image regardless of its class, so it converges within a few dozen samples and depends only weakly on the model's own possibly wrong pseudo-labels. The class residuals start out tightly constrained, and their prior loosens automatically as evidence for each class accumulates. This turns two separate heuristics into one posterior update: mean-subtraction tricks for the gap, and a fixed prior-versus-data weight in online adaptation.

Core Claim

Online Gaussian Test-Time Adaptation of Vision-Language Models (OGA) models each class's visual features as a Gaussian, maintains a cache of confident test samples, and uses maximum a posteriori (MAP) estimation to fuse the zero-shot text prior with these streaming statistics. It reports +16.2% over zero-shot CLIP on EuroSAT with one hyperparameter setting across 11 benchmarks. Its prior places class means at the text embeddings, as if both modalities lived on the same manifold. Mind the Gap shows that this is false by construction: architecture and contrastive training keep the two modalities apart by a stable offset.

Our claim is that a hierarchical prior over class means improves both average accuracy and worst-case accuracy over OGA. The prior separates a shared, label-free gap term from class-specific residuals, and it has a covariance taken from source data. The claim is also that it improves over OGA combined with simple centering. The second comparison is the one that matters.

Model

Let $x$ be a normalized CLIP image embedding of a test sample, $t_k$ the text embedding of class $k$, and $\mu_k$ the unknown mean of class-$k$ image embeddings in the target domain. The prior is:

$$ \mu_k = t_k + g + r_k,\qquad g \sim \mathcal{N}(g_0, S_g),\qquad r_k \sim \mathcal{N}(0, S_r). $$

Here $g$ is the shared modality displacement and $r_k$ is the class residual. The quantities $g_0$, $S_g$ and $S_r$ are estimated offline from paired image-caption data such as MSCOCO:

  • $g_0$ is the mean image-minus-caption difference.
  • $S_r$ is the scatter of per-pair differences once $g_0$ is removed.
  • $S_g$ is the variation of $g_0$ across subsets of the pairs.

Test features follow $x \mid k \sim \mathcal{N}(\mu_k, \Sigma)$, with $\Sigma$ estimated by the same Bayes Ridge estimator (a shrinkage covariance estimate that stays invertible with few samples) OGA uses.

Inference runs online by alternating two steps:

  • Class responsibilities. Each new sample gets soft class assignments under the current posterior.
  • Closed-form Gaussian updates. The posterior over $g$ pools all samples. The posterior over each $r_k$ uses only that class's soft-assigned samples.

The posterior precision of $r_k$ grows with the effective count of class $k$. The balance between prior and data therefore changes as evidence arrives, which addresses OGA's fixed weighting directly.

This model also explains the apparent contradiction between Mind the Gap, which finds a non-zero gap can be optimal, and Closing the Modality Gap Aligns Group-Wise Semantics and I0T, which drive it to zero. Under our model, classification depends on where each $\mu_k$ actually lies, not on the gap's size. A global shift and a rescaling of the softmax interact in ways that can make a nonzero offset look optimal under a fixed temperature. We test this reading directly in the analysis below.

Relation to Prior Work

Three lines of work supply the ingredients:

  • Informative priors from pretraining. Pre-Train Your Loss shows that a source-derived Gaussian with low-rank-plus-diagonal covariance beats SGD fine-tuning as a downstream prior. Its reported example is 74.15 versus 73.17 mIoU on Pascal VOC. We borrow the principle of a covariance measured on source data, but apply it to a few vectors in embedding space rather than millions of weights. That sidesteps the cost problems of that line.
  • Deterministic gap removal. GR-CLIP and I0T_post remove the gap by subtracting modality means. GR-CLIP reports up to 26 NDCG@10 points on MixBench. I0T_post moves Flickr30k I2T recall from 69.60% to 73.30% and T2I from 67.10% to 76.30%. Both treat the gap as a fixed translation with no uncertainty and no class structure. Our $g$ term is a probabilistic version of their correction, and $r_k$ is what they omit.
  • Gap correction under shift. TCR shows that the gap widens under query shift and counters it by tuning LayerNorm with an empirically estimated target gap. This motivates tracking $g$ online rather than fixing it. Unlike TCR, we leave the network untouched.

ViPMoG corrects image-image versus image-text score mismatch with a fixed affine map. That is another deterministic stand-in for what $g$ and $S_r$ model.

Research Program

The work has three parts:

  1. Offline prior. Estimate $g_0$, $S_g$ and $S_r$ from MSCOCO pairs. Because $S_r$ is a full 512- or 768-dimensional matrix, we test diagonal, low-rank-plus-diagonal, and full versions with shrinkage.
  2. Online algorithm. Implement the alternating posterior update inside OGA's codebase, keeping OGA's cache and covariance estimator, so that the only change is the prior.
  3. Gap-optimum analysis. For CIFAR-10, sweep a global shift and the softmax temperature jointly. This tests whether the 0.9013→0.9081 improvement in Mind the Gap disappears once temperature is re-tuned, as our reading predicts.

Baselines

Zero-shot CLIP. The naive reference is zero-shot CLIP with standard prompts on the same backbone, which does no adaptation.

OGA. The strongest existing method is OGA, run from its released settings with its single fixed hyperparameter set. On EuroSAT, OGA reports +16.2% over zero-shot CLIP. The file gives no absolute per-dataset accuracies or tail-accuracy numbers, so we will reproduce them ourselves.

OGA with GR-CLIP-style centering. This is the most dangerous competitor. It estimates image and text means from a small calibration set, centers both modalities, and then runs unmodified OGA. We also run a variant that updates the image mean as a running average over the stream.

Network-tuning reference. TCR-style LayerNorm tuning with its gap objective, applied to classification, serves as the reference for methods that update the network itself.

Ablations. Four ablations isolate the key ingredient:

  • shared $g$ only, with $r_k$ removed;
  • $r_k$ only, with $g$ fixed at zero;
  • isotropic $S_r$ in place of the source-derived covariance;
  • a $g$ update restricted to confident cached samples, to measure how much the label-free pooled update reduces confirmation bias.

Fairness and success criterion. Every method uses the same backbone (ViT-B/16), the same streams with five random orderings, and no per-dataset tuning. Our own prior is fixed once from MSCOCO and never touched on target data. Beating OGA means a higher mean accuracy over the 11-dataset suite with non-overlapping confidence intervals across orderings, without lowering Expected Tail Accuracy (the mean accuracy of the worst 10% of runs). Beating centering-plus-OGA by a margin larger than the spread across orderings is required before we credit the hierarchical structure.

Why not just…?

Why not just center the embeddings and then run OGA? This is a real threat, because centering removes most of the mean offset at almost no cost. Its known weaknesses are these. It needs a batch to estimate means, as the I0T authors note. It applies one rigid shift everywhere. Under shift, the gap moves, as TCR documents. Whether these weaknesses matter in practice is an empirical question. The ablation "shared $g$ only" is close to a principled running-mean centering. If it matches the full model, the residual covariance contributes nothing. We would then reframe the work as uncertainty-weighted streaming centering, and report that honestly.

Why not just schedule OGA's prior weight by sample count? A count-based schedule addresses fixed weighting but still centers the prior at the wrong location. We include a count-scheduled OGA in which the weight grows as $n/(n+c)$, with $c$ fixed globally, the form a conjugate Gaussian update would give. If it closes most of the gap to our method, the benefit came from dynamic weighting rather than gap modeling.

Why not just tune LayerNorm as TCR does? TCR changes the network through gradient steps and relies on a confident-pair heuristic for the target gap. That costs backpropagation per batch and risks drift on long streams. We compare accuracy, per-sample latency, and stability over the stream. A training-free method that matches TCR's accuracy at a fraction of the cost is a meaningful result. A clear TCR win would mean shifts in the representation itself, not just the gap, dominate.

Decisive Checks

Main benchmark. We run on OGA's 11-dataset suite, including EuroSAT, DTD, Oxford Flowers-102 and Oxford-IIIT Pets. Metrics are top-1 accuracy, Expected Tail Accuracy, and expected calibration error.

Mechanism check. On held-out labeled data, we measure the distance between each posterior class mean and the true class mean of image embeddings, over the course of the stream. The full model should reach a given error with fewer samples per class than OGA or centering-plus-OGA, especially for rare classes, where pooled estimation of $g$ should matter most. If the advantage does not concentrate on rare classes and early in the stream, the hierarchical explanation is wrong, even if accuracy improves.

Robustness to source mismatch. We estimate the prior from Flickr30k instead of MSCOCO and check that results hold.