Few-Label Audits of Zero-Shot Models Under Distribution Shift

When a vision-language model such as CLIP is deployed zero-shot, meaning with no task-specific training, on a shifted population (new hospitals, adversarially hard photos, birds on unusual backgrounds), nobody knows how accurate it is there until someone labels data from that population. Labels are the expensive part. This proposal asks how few labels suffice to estimate a zero-shot model's accuracy, and especially its accuracy on the worst-off subgroup, on a shifted target pool without bias. The approach is to let a cheap automatic "proxy" guess the model's loss on every unlabeled example and to spend human labels only on correcting the proxy's mistakes. Our central bet concerns the proxy itself: the obvious proxy, the evaluated model's own confidence, fails exactly under shift. A deliberately debiased proxy, combined with a labeling rule that seeks out sparse regions of the embedding space, should shrink the labels needed by a large factor.

The opportunity comes from two literatures that have not met. Prediction-powered active testing (PPAT) gives an estimator of model risk that stays unbiased however bad the proxy is. It has only been tested on in-distribution tabular regression. Meanwhile, work on zero-shot models under shift shows that these models are often confidently wrong on the inputs that matter (RLCF, SEraser). A proxy that is confidently wrong in the same places as the model predicts low loss precisely where true loss is high. Unbiasedness survives this, but the variance savings that justify using a proxy do not.

Core Claim

The variance of a control-variate risk estimator is set by how well the proxy's predicted loss tracks the true loss on the target pool. Under spurious-correlation and natural-adversarial shift, a proxy that shares the evaluated model's shortcuts tracks true loss worst on the rare, shortcut-conflicting examples that dominate worst-group error. We claim that two changes restore most of the savings. First, build the proxy from geometric debiasing tools: translate image embeddings away from text-described spurious directions as in TIE, and penalize scores of images lying in sparse parts of a group's embedding cloud as in DAT. Second, have the acquisition rule draw labels preferentially where the debiased proxy and the evaluated model disagree, or where local density is low. The new contribution is not a better classifier. It is the observation that debiasing methods, which were built to improve zero-shot predictions, are most valuable as auditors of zero-shot predictions. Their errors are decorrelated from the model's shortcut errors, and that decorrelation is what matters for variance.

Concretely, we use the PPAT estimator. Let a pool of $N$ unlabeled target inputs $x_i$ be given. A proxy produces predicted loss $\hat\ell(x_i)$, and $n \ll N$ inputs $x_j$ are drawn from an acquisition distribution $q$ and labeled to reveal the true loss $\ell(x_j)$. The estimator is

$$\hat R = \frac{\lambda}{N}\sum_{i=1}^{N}\hat\ell(x_i) + \frac{1}{n}\sum_{j=1}^{n} \frac{1}{N q(x_j)}\big(\ell(x_j)-\lambda\,\hat\ell(x_j)\big).$$

Here $\lambda$ sets how much the proxy is trusted. The estimator is unbiased for any proxy. Its variance scales with the spread of the weighted residuals $\ell-\lambda\hat\ell$, so a proxy whose errors are uncorrelated with the model's shortcut errors directly lowers it. Worst-group accuracy is estimated by applying the same estimator within groups. Groups are recorded for labeled points by annotators and inferred for unlabeled points by the proxy, as in TIE* and DAT*.

Research Program

We first characterize the failure. We take the evaluated systems as given: zero-shot CLIP ViT-B/16 and ViT-L/14, TPT, DynaPrompt, BaFTA, TIE, DAT, and naive Online TPT, which DynaPrompt reports collapsing to 6.96% on ImageNet-A. For each system, we measure the correlation between the self-proxy (one minus the model's own max probability) and true loss, separately on majority and minority groups. We expect this correlation to be weak or negative on minority groups. That would turn the confident-misprediction findings of RLCF and SEraser into a statement about evaluation variance.

Second, we build three proxy families of increasing independence from the evaluated model. The first is the model itself. The second is a different-architecture zero-shot model, which tests the concern from Natural Adversarial Examples that failures transfer across architectures. The third is a debiased proxy that combines TIE translation and DAT density scores on top of a CLIP encoder. Proxy loss is the expected loss under the proxy's predictive distribution over labels, scored against the evaluated model's prediction.

Third, we design the acquisition rule. We mix PPAT's residual-uncertainty sampling with two shift-aware terms: model–proxy disagreement, and inverse local density measured by the simplified local outlier factor (SLOF) used in DAT. The mixture is always defensive, meaning it keeps a uniform component so that importance weights stay bounded. A small theoretical piece bounds the estimator's variance in terms of the proxy–model error correlation within each group. This makes precise when a shared-shortcut proxy cannot help, which is the gap left open by the RLCF circularity concern.

The guiding constraint comes from Taori et al.. They show that conclusions about zero-shot robustness depend heavily on the reference distribution. We therefore estimate target accuracy directly with labels from the target pool, rather than predicting it from in-distribution accuracy.

Baselines

The naive reference point is uniform random labeling. We draw $n$ target examples at random, label them, and report their mean accuracy and per-group means. Its cost is easy to state: the binomial standard error at 50% accuracy with 200 labels is about 3.5 points, and it is far worse within small minority groups. It is run at the same label budgets as every other method.

The label-free reference point is the evaluated model's average confidence used as its accuracy estimate. This costs zero labels and is what practitioners often report. The evidence that confidence is miscalibrated under shift (RLCF, SEraser) predicts it fails, and we will show by how much.

The strongest existing method is PPAT (Ashouritaklimi et al.) with its residual-based acquisition and tuned $\lambda$, using the evaluated model as proxy. We also run PPAT with our debiased proxy, so that proxy and acquisition contributions separate. There is no reported result found on Lacuna for PPAT, or for any active-testing method, on Waterbirds, CelebA, ISIC, or ImageNet-A. PPAT reports only qualitatively "lower variance and tighter bounds" than LURE and random sampling on UCI regression. We therefore reimplement it with identical pools, budgets, random seeds, and the same $\lambda$-tuning procedure as our method. We also include LURE importance-sampled active testing without a proxy, and a strong simple alternative: stratified random labeling over proxy-inferred groups.

Two ablations isolate the key ingredients: debiased proxy with uniform acquisition, and self-proxy with density-and-disagreement acquisition. Beating the baselines means reaching a target error (absolute error under 2 points on overall accuracy, under 4 points on worst-group accuracy, with 95% interval coverage near nominal) with at least 2x fewer labels than PPAT-with-self-proxy and stratified sampling, on at least the group-shift benchmarks.

Why not just…?

Why not just label a few hundred random examples? For overall accuracy on a pool like ImageNet-V2, this may be nearly enough, and we will report where it is. Worst-group accuracy is the binding case. On Waterbirds, the gap between vanilla CLIP at 31.93% worst-group accuracy and DAT at 83.33% (DAT) lives in a minority group that random sampling rarely hits. Small differences between methods, such as TIE's 78.82% versus DAT's 83.33%, are unresolvable without many minority labels. Our learning curves show the budget at which random sampling correctly ranks these systems.

Why not just trust the model's own confidence and skip labels? Confidence is the quantity that shift corrupts. RLCF and SEraser both document models becoming more confident in shortcut-driven errors. The label-free baseline makes this concrete, and our main experiment (described below) is designed to show it reporting high accuracy for a collapsed model.

Why not just run PPAT unchanged? This is the real threat. PPAT is already unbiased, and its tuned $\lambda$ partly protects against a bad proxy by down-weighting it. If $\lambda$-tuning alone recovers most of the savings, our contribution shrinks to an analysis result. The self-proxy-versus-debiased-proxy comparison under identical acquisition settles this directly. We commit to reporting it even if the gap is small.

Why not just stratify labels by proxy-inferred group? Stratification is simple and may capture much of the benefit. It fails when the proxy mislabels the spurious attribute, the documented weakness of TIE*. It also fails when the shortcut is non-semantic, as with ISIC marker patches or X-ray hospital artifacts. The density term in our acquisition rule does not need a nameable confounder, and the medical benchmarks test exactly this.

Proposed Experiments

We simulate label budgets on fully labeled shifted pools: Waterbirds, CelebA, ISIC, and the COVID-19 chest X-ray benchmark for group shift; ImageNet-A, -R, -V2, and -Sketch for natural shift; CIFAR-10-C for corruption. For each of the evaluated systems, at budgets from 25 to 1,000 labels and over 200 seeds, we report root-mean-square error and interval coverage for overall and worst-group accuracy. We also report the probability of correctly ranking pairs of systems.

The decisive check is collapse detection. Online TPT on ImageNet-A drops from 54.77% to 6.96% (DynaPrompt). The self-proxy, being confident, should report high accuracy, while our estimator should flag the collapse within roughly 50 labels. A second check is the medical datasets, whose non-semantic shortcuts break text-direction debiasing. If the debiased proxy loses its advantage there while density-driven acquisition retains it, that cleanly separates the two ingredients. If neither ingredient beats tuned PPAT with stratification on any benchmark, the conclusion is that existing machinery already suffices, and we will say so.