A vision-language model such as CLIP classifies an image by comparing it with text: it embeds a sentence like "a photo of a zebra" and picks the class whose sentence lies closest to the image. Richer class descriptions help, for example "has black and white stripes" or "a horse-like animal with a short mane". But whether a given description is useful depends on where the image comes from. Stripes identify a zebra in a photo. They are useless in a one-color doodle, where only body shape remains. Today these descriptions are fixed before deployment and weighted equally no matter what the test images look like.
We propose to adapt the meaning of the labels at test time instead of the image features or the network weights. Each class is described along the same small set of aspects, such as shape, color, texture, typical setting and parts. A shared low-dimensional weight vector over those aspects, plus a few rendering-style phrases ("a sketch of", "a painting of"), is searched online on the unlabeled test stream with a derivative-free optimizer. The frozen model is never updated. The adapted state is a short, human-readable vector ("in this domain, trust shape and ignore color"). Because the search happens entirely on the text side, the image encoder runs once per image. New classes need only aspect descriptions, so the method stays open-vocabulary.
Core Claim
Adapting in a small latent space works when that space holds the directions the shift needs. Semantic label embeddings can supply such a space, and it can be named and interpreted. Our specific claim is that on shifts in visual domain, a low-dimensional test-time search over aspect weights of label descriptions will beat two alternatives. The first is uniform description averaging, the static semantic baseline. The second is label-blind feature-subspace correction, the latent-adaptation baseline. We expect the largest gains on abstract domains such as sketches and doodles, where some aspects stop carrying information. We further claim the recovered weights will be predictable from the domain: color and texture should be down-weighted on quickdraw, for example. A random subspace of the same dimension should not reproduce the gain.
Concretely, let $f(x)$ be the frozen image embedding of input $x$. Let $t_{c,k}$ be the frozen text embedding of class $c$ described along aspect $k$ (with $k = 1..K$, $K \approx 8$). Let $d_j$ be the embedding of style phrase $j$ (with $j = 1..J$, $J \approx 6$). The class score is
$$s_c(x; w, u) = \cos\Big(f(x),\ \sum_{k} w_k\, t_{c,k} + \sum_{j} u_j\, d_j\Big),$$
where $w$ lies on the probability simplex over aspects and $u$ is a small nonnegative style mixture. Both are shared across all classes. At test time we choose $(w,u)$ to minimize
$$\mathcal{L}(w,u) = \mathbb{E}_{x \in B}\, H\big(p(\cdot \mid x)\big) - \lambda\, H\big(\bar p\big),$$
where $B$ is the current test batch or a sliding window of the stream, $p(\cdot\mid x)$ is the softmax over scores $s_c$, $H$ is Shannon entropy, and $\bar p$ is a running average of predictions over the stream. The second term penalizes collapse onto a few classes.
Relation to Prior Work
Three latent-adaptation papers supply the mechanism. ELaTTA builds a principal subspace of backbone features with truncated SVD. It then searches a correction vector with CMA-ES (an evolution strategy that needs only forward evaluations) to minimize prediction entropy. It is efficient, but it knows nothing about label semantics, is tied to a closed classification head, and optimizes an objective the file flags as collapse-prone. We reuse its derivative-free search but move it to the label side, inside a subspace defined by meaning. Gradient-based inference of task representations infers a task vector by backpropagating prediction error into a dedicated layer of a frozen network. That vector is uninterpretable. Our aspect weights play the same role but each coordinate has a name, and we test a one-gradient-step variant alongside CMA-ES. Multi-Prompt Alignment is the closest existing work. It compresses learned soft prompts for frozen CLIP into an autoencoder latent subspace and adapts new domains by tuning coordinates there. Its latent coordinates are unnamed. Adaptation takes hours (11 hours on an RTX 3090 per the paper), and the class set is fixed across domains. Our subspace is built from text, adaptation runs online in seconds, and unseen classes are allowed.
Two zero-shot domain generalization papers motivate the domain-context term and frame the conflict we try to resolve. Structured Latent Embeddings enforces domain invariance around attribute centroids. COCOA argues that domain information should instead be modeled explicitly, conditioning batch normalization on attributes plus a domain embedding. At test time, though, COCOA only mixes known source-domain embeddings heuristically. Our style weights $u$ are a searched, test-time version of COCOA's domain context. The aspect weights $w$ implement a soft form of invariance: they keep only the aspects that survive the shift.
The semantic-label papers show that descriptions help and also where they break. DELE needs heavy bidirectional attention to filter irrelevant WordNet senses. Label-description training shows that curated descriptions raise zero-shot accuracy substantially. StructCBM organizes labels into prototypical and discriminative concepts, refined offline by an LLM. Gen-Z scores inputs against contextualized label descriptions but assumes a uniform class prior. That assumption is why we add the marginal term and track the running average $\bar p$ instead of fixing it at uniform. All four fix the description weighting before test time. Our proposal treats that weighting as the quantity to adapt. We keep the per-aspect structure shared across classes so the search dimension stays constant, which avoids StructCBM's quadratic growth in class pairs.
Research Program
We will build an aspect bank for each benchmark's class vocabulary. An LLM writes one short description per class per aspect from a fixed template, such as "the shape of a {class}: …". Each description is embedded once with the frozen CLIP text encoder and cached. The style bank is a handful of rendering phrases, chosen without looking at target domains. At test time, each image is encoded once. Every CMA-ES candidate $(w,u)$ then costs one weighted sum of cached text vectors and one matrix product, so the search cost does not grow with the size of the image encoder.
The analysis has three parts. The first is accuracy under shift, on both seen and unseen classes. The second is whether the recovered $w$ matches intuition and is stable across seeds; we compare against weights fit by an oracle on labeled target data. The third is failure modes: collapse, degradation on clean data, and sensitivity to window size and to $\lambda$. We will also test the gradient variant, one backpropagation step into $(w,u)$ in the style of the gradient-inference paper, to see whether forward-only search is actually needed.
Baselines
The naive reference is frozen CLIP with the plain class-name prompt, with no descriptions and no adaptation. Just above it is the static semantic baseline: uniform averaging over all aspect descriptions and style phrases. This baseline is the direct test of our claim, because it uses exactly our text bank and differs only in not adapting. Beating it requires a gain on the target domain beyond seed variance across three description banks generated independently.
The strongest label-blind latent competitor is ELaTTA (paper). We re-run it on the same CLIP image encoder, using the uniform-average text classifier as its head. We give it the same CMA-ES budget in function evaluations and the same stream window. On ImageNet-C with a supervised ViT-Base, ELaTTA reports 57.82% average accuracy, against 55.03% unadapted and 56.34% for MEMO. That setting uses a different backbone, so we treat those figures as context and compare only against our re-run. We will also combine ELaTTA's feature correction with our label-side search, to test whether the two are complementary.
The strongest existing CLIP latent-subspace method is Multi-Prompt Alignment (paper). It reports 54.1% average accuracy on DomainNet multi-source adaptation. That result is closed-set, uses unlabeled target data offline, and takes about 11 hours. We run it on our DomainNet and Office-Home splits with its published protocol. We report accuracy together with adaptation wall-clock time. The file gives no Office-Home accuracy for it, only its 2.36M tunable parameters. We do not expect to beat it on closed-set accuracy. A win means coming within a small margin at orders of magnitude less adaptation time, and handling unseen classes, which it cannot do.
For zero-shot recognition of unseen classes in unseen domains, the file reports COCOA at 22.6% DomainNet average and 13.1% on quickdraw, and Structured Latent Embeddings at 21.9% and 11.5%. These use non-CLIP backbones and word-vector attributes. We cite them as the reported task baselines and do not claim a like-for-like win over them.
Two ablations isolate the key ingredient. The random-subspace control replaces the $K$ aspect directions per class with $K$ random directions in text-embedding space, projected into the span of each class's embeddings. Search dimension and search budget are unchanged. The per-domain oracle fits static $(w,u)$ on labeled target data. It is an upper bound for any fixed weighting and shows how much of the oracle gain test-time search recovers.
Why not just…?
Why not just average all the descriptions and skip adaptation? This is the most serious threat. If the aspect weights that work best are close to uniform in every domain, our method collapses to this baseline. The per-domain oracle ablation settles it. If oracle weights beat uniform averaging by very little on quickdraw and sketch, the premise is wrong and we will report that. If the gap is large, the question becomes how much of it our unlabeled search closes.
Why not just run ELaTTA-style feature correction on CLIP? Feature correction can move an image embedding toward a confident class. But its subspace is fit on source images, and the file names this as the main limitation of such methods: the shift may lie outside the span of source-domain features. Aspect reweighting instead changes which evidence counts, and its coordinates are the same for seen and unseen classes. The head-to-head re-run, broken down by domain and by seen versus unseen classes, will show which approach helps where. The combined variant will show whether they overlap.
Why not just fine-tune with LoRA or tune soft prompts on the target domain? That approach needs labels or long unsupervised optimization, and it gives up open vocabulary. The file reports the cost: Multi-Prompt Alignment needs hours even with its latent-subspace shortcut, and gradient-based test-time methods such as MEMO exhaust memory on edge devices (ELaTTA). Where labels are available, prompt tuning will likely win on closed-set accuracy. Our claim is limited to the online, unlabeled, open-vocabulary regime, and the Multi-Prompt Alignment comparison reports both accuracy and time.
Why not just ask an LLM to pick relevant descriptions per domain? An LLM can guess that doodles lack color. It cannot see the test stream, and calling it per input is what StructCBM was designed to avoid (paper). We will include a variant in which an LLM is told the domain name and outputs aspect weights once per domain. If it matches our searched weights, the search is unnecessary whenever the domain is known in advance. The method would then matter only for unnamed or drifting shifts, which we test with mixed corruption streams.
Proposed Experiments
Experiments use DomainNet in two protocols: the unseen-class, unseen-domain split used by COCOA and Structured Latent Embeddings, and the multi-source closed-set protocol of Multi-Prompt Alignment. We add Office-Home, and the ImageNet-C, ImageNet-R and ImageNet-Sketch suites evaluated by ELaTTA. The file gives no ELaTTA numbers for ImageNet-R or ImageNet-Sketch, so all results on those come from our re-runs. Metrics are top-1 accuracy (harmonic mean of seen and unseen classes on the zero-shot split), expected calibration error, forward evaluations and wall-clock time per adapted batch, and clean-accuracy change on unshifted ImageNet.
The decisive checks are four. First, adaptive aspect weighting must beat uniform averaging on the abstract domains (quickdraw, sketch) of the zero-shot split while not hurting clean accuracy. Second, the random-subspace control must fail to reproduce that gain. If it succeeds, the benefit comes from low-dimensional label-side search in general, not from semantic structure. Third, recovered weights must correlate across seeds and with oracle weights, and must down-weight color and texture on quickdraw. Fourth, on unseen classes the gain must hold even though those classes played no part in building the aspect bank's template. If the first check fails, the idea fails. If only the third fails, the method may still work but the interpretability claim is withdrawn.
References
- Efficient Edge Test-Time Adaptation via Latent Feature Coordinate Correction
- Gradient-based inference of abstract task representations for generalization in neural networks
- Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation
- Structured Latent Embeddings for Recognizing Unseen Classes in Unseen Domains
- COCOA: Context-Conditional Adaptation for Recognizing Unseen Classes in Unseen Domains
- Description-Enhanced Label Embedding Contrastive Learning for Text Classification
- The Benefits of Label-Description Training for Zero-Shot Text Classification
- Adaptive Concept Discovery for Interpretable Few-Shot Text Classification
- Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions