Selecting Robust Models with a Handful of Group Labels

Methods that stop classifiers from relying on spurious cues, such as backgrounds, hospital markers, or rulers next to skin lesions, are often described as "group-free" because they need no group labels during training. In practice they still need them. A "group" here is a combination of class and nuisance attribute, such as "waterbird on land." Hyperparameters and stopping points for these methods are chosen by measuring worst-group accuracy on a validation set where every image carries a group label. Simple data balancing achieves competitive worst-group-accuracy found that this validation access, rather than the training algorithm, is the main driver of reported robustness. Contrastive Adapters for Foundation Model Group Robustness lists the same dependence as a limitation. In label-scarce domains such as dermatology and radiology, a fully group-labeled validation set can cost more expert time than the training labels themselves. That makes model selection the hidden bottleneck of label-efficient robustness.

We propose to treat model selection as a small-sample estimation problem and solve it with a pretrained encoder. The procedure has three steps. First, an expert labels the nuisance attribute on a few examples per group. Second, an attribute classifier built from those examples on a frozen foundation encoder assigns pseudo-attributes to the much larger validation pool, which needs only class labels. Third, the few true labels are used to measure and invert the classifier's confusion. The result is a debiased estimate of each candidate model's worst-group accuracy, with far lower variance than the few labels could give alone. The claim is that this estimator selects models whose true worst-group accuracy comes close to what full group-labeled validation achieves, at a few percent of the annotation cost. The comparison that matters is against simply using those same few labels directly.

Core Claim

Worst-group accuracy measured on $m$ labeled examples of the rarest group has a standard error of roughly $\sqrt{p(1-p)/m}$. This is the binomial standard error of an accuracy $p$. With $m=10$ and $p=0.7$ it is about 0.14. That is large enough that selection among a grid of runs is close to random whenever candidates differ by less than about ten points, which is the usual regime. Pseudo-labeling the whole validation pool removes most of that variance but introduces bias, because the attribute classifier makes errors.

Our estimator spends the few true labels on correcting that bias rather than on estimating accuracy directly. Work within one class $y$. Let $\tilde a$ be the vector of the candidate model's accuracies on validation examples grouped by predicted attribute. Let $a$ be the vector of accuracies on true attribute groups. Let $M$ be the matrix whose entry $M_{jk}$ is the probability that an example with predicted attribute $j$ truly has attribute $k$, estimated from the labeled examples. If the attribute classifier's errors are independent of the candidate model's correctness, given the class and the true attribute, then

$$\tilde a = M a, \qquad \hat a = M^{-1}\tilde a,$$

and worst-group accuracy is estimated as the minimum of $\hat a$ over attributes and classes.

We select by a lower confidence bound on that minimum rather than by the point estimate. The bound is obtained by bootstrapping both the labeled set and the pool, which protects against picking a lucky noisy candidate. The conditional-independence assumption is the method's weak point: a model that fails on the minority group may fail on exactly the images the attribute classifier also gets wrong. We therefore test it directly, as described below, rather than assume it.

This differs from the closest prior work in where the scarce labels are spent. SpurCon also annotates ten examples per group and pseudo-labels spurious attributes through prototypes on frozen BiomedCLIP embeddings. It uses those pseudo-labels to weight a training loss and never addresses how its own runs are selected. SLIM spends human feedback on under 3% of training images to curate a balanced training subset. Again the labels go into training. Our proposal spends the labels on evaluation and corrects the pseudo-labels for bias. Neither paper does either. Because the estimator selects among candidates, it applies on top of any training method, SpurCon and SLIM included.

Research Program

The work has three parts.

The first is the estimator itself. We build the attribute classifier as nearest-prototype classification in the embedding space of a frozen encoder: CLIP for natural images and BiomedCLIP for medical images, matching SpurCon's setup. Prototypes come from the $m$ labeled examples. Confusion matrices are estimated per class with additive smoothing, so that inverting them stays stable when $m$ is tiny. We use the corrected estimate only when the estimated $M$ is well conditioned. Otherwise the procedure falls back to the uncorrected pseudo-group estimate and reports the fallback.

The second is a stress test of the independence assumption. On benchmarks with full group labels, we measure the gap between the corrected estimate and true validation worst-group accuracy for every candidate. We then relate that gap to the overlap between the candidate's errors and the attribute classifier's errors. This tells us when the method fails, not only whether it works on average.

The third is a fair annotation-budget accounting. For each total budget of expert group labels, we compare two options. One spends the budget on validation through our estimator. The other spends it on training supervision, for example SpurCon-style pseudo-labeling, and selects by average accuracy. This answers the practical question of where a fixed number of expert labels does the most good.

The candidate pool spans training methods and their hyperparameter grids. It includes group-balanced subsampling, as in Simple data balancing achieves competitive worst-group-accuracy, and contrastive adapters on CLIP, as in Contrastive Adapters for Foundation Model Group Robustness. It also includes SpurCon's weighted contrastive head and standard cross-entropy linear probes. Every method contributes roughly the same number of configurations and checkpoints, so that no single method dominates the pool.

Baselines

The naive reference point is selection by average validation accuracy, which needs only class labels. This is the "do nothing" default. It is expected to favor runs that exploit the shortcut, since the shortcut remains predictive for the majority group. Bridging Explainability and Embeddings: BEE Aware of Spuriousness notes exactly this tension between average and worst-group accuracy. A close variant selects the last checkpoint of each configuration. Both are run over the identical candidate pool.

The strongest competitor uses the same $m$ group labels per group to compute worst-group validation accuracy directly, then selects the maximum or the lower bound. It costs exactly the same annotation, needs no encoder, and is what a careful practitioner would do. Our method beats it only if, at equal $m$, the selected model's true test worst-group accuracy is higher. We require the improvement to be statistically reliable across at least 20 random draws of the labeled subset at $m \in \{5, 10, 20\}$. We expect the advantage to shrink as $m$ grows. Identifying the crossover point is itself a result.

The upper reference is selection with the full group-labeled validation set, which is the protocol behind published numbers. On this protocol the file reports the following worst-group accuracy (WGA) results: 90.0% on CelebA for contrastive adapters (Contrastive Adapters for Foundation Model Group Robustness); 91.0% on Waterbirds and 79.1% on ISIC for SLIM (SLIM); and 73.0% on CheXpert and 65.0% on ISIC 2020 for SpurCon, against baselines of 35.6% and 0% (SpurCon). For simple subsampling the file contains no reported absolute WGA numbers on Lacuna, only training-time comparisons. We reproduce oracle selection on our own candidate pool rather than copying these figures. We report "selection regret," defined as oracle WGA minus selected-model WGA, so that differences in training pipelines do not contaminate the comparison. Recovering at least 80% of the gap between average-accuracy selection and oracle selection at $m=10$ counts as success.

Two ablations isolate the key ingredient. The first uses pseudo-groups without the $M^{-1}$ correction, which isolates the value of bias correction. The second is a fully label-free version: per-class k-means clusters in encoder space with worst-cluster accuracy as the selection score. This isolates the value of the few attribute labels at all. All variants share the same encoder, candidate pool, and validation pool, and none is tuned on test data.

Why not just…?

Why not just use the few group labels directly as a tiny validation set? This is the real threat. If candidates differ by large margins, ten labels per group may be enough, and our estimator adds complexity for nothing. The binomial variance argument predicts that direct estimation fails exactly when candidates are close. But that is a prediction, not evidence. The equal-budget comparison across repeated label draws settles it, and we will report the result even if direct estimation wins at $m \geq 20$.

Why not just select by average accuracy, or use the last checkpoint? Average accuracy rewards the shortcut. Evidence that this matters comes from Simple data balancing achieves competitive worst-group-accuracy: balancing methods without group-labeled validation become unstable, and reweighting in particular degrades if training runs too long. Our regret measurements quantify how much average-accuracy selection loses on each benchmark. If that loss turns out to be small somewhere, that benchmark does not need the method.

Why not just spend the labels on training, as SpurCon does, and skip careful selection? SpurCon shows that ten labels per group yield large training gains. Even so, its own runs still need some selection rule, and the file gives no evidence about how it behaves without group-labeled validation. The budget-allocation experiment compares these uses of labels directly. A plausible outcome is that the best use splits the budget, using one prototype set for both training and evaluation. Reusing labels this way risks optimistic bias, so we will also test it with disjoint label sets.

Why not just discover groups without any labels, through clustering or learning speed? The label-free cluster ablation tests clustering. Signals based on learning speed, as in SEBRA and Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling, assume shortcuts are learned faster than core features. Evaluating and Improving Robustness of Self-Supervised Representations to Spurious Correlations reports that this assumption broke on MetaShift, where the background was harder to learn than the pet. We include MetaShift precisely to show whether label-free selection fails where a few attribute labels do not.

Proposed Experiments

We evaluate on five benchmarks with group annotations: Waterbirds, CelebA (blond hair with gender as the spurious attribute), MetaShift, ISIC with ruler and patch markers, and CheXpert pneumothorax with chest drains as the spurious cue. The last two follow SpurCon's definitions. CivilComments is an optional text check that would use a sentence encoder.

For each benchmark we hold the candidate pool fixed. We then report four quantities for every selection rule at $m \in \{5, 10, 20, 50\}$, averaged over at least 20 label draws: test WGA of the selected model, selection regret, Kendall rank correlation between estimated and true validation WGA across candidates, and coverage of the lower confidence bound.

Three checks are decisive. The first is whether the corrected estimator beats direct small-sample estimation at $m=10$ on at least four of the five benchmarks. The second is whether the correction step beats uncorrected pseudo-groups, particularly on ISIC and CheXpert, where frozen-encoder attribute recognition is weakest. The third is whether the independence diagnostic predicts the benchmarks where the estimator is biased.

A negative result on the first check, with direct estimation matching ours, would still be useful. It would show that roughly ten labels per group suffice for selection, which would shrink the validation-oracle problem to a modest and quantified annotation cost.