Tag-and-Count Validation for Worst-Group Model Selection

Models trained to be robust to spurious correlations are usually judged, and chosen, by their accuracy on the rarest subgroup of a validation set. The rarest subgroup is rare there too. In a typical image benchmark, the decision about which training run and which epoch to ship rests on a few dozen to a couple hundred examples. Their binomial noise is often larger than the real differences between candidate models. Picking the maximum over many noisy candidates then produces the familiar collapse from validation to test. We propose to borrow a measurement trick from particle physics, double-tag reconstruction, to estimate subgroup accuracy from the whole validation set instead of only from the few examples someone has hand-labeled with their group.

In double-tag measurements, one particle of a pair is identified (it serves as a "tag" that heralds its partner's presence). The physicist then counts how often the partner decays a particular way. The total number of pairs produced cancels out of the ratio. Our analogue works as follows. A cheap, imperfect group detector, independent of the models being compared, tags every validation example with a guessed subgroup. A small hand-labeled calibration set measures how often that detector confuses groups. A joint likelihood fit over tag channels then recovers each candidate model's per-group accuracy, and the unknown group sizes cancel just as the production count does. Two consequences follow. With a fixed annotation budget, the minority accuracy estimate uses every validation example rather than a few dozen, so selection noise drops. And because the calibration error is shared across candidates, it largely cancels when two checkpoints are compared.

Core Claim

At a fixed group-annotation budget of 25 to 200 labels, a tag-and-count estimator should select checkpoints whose test worst-group accuracy is closer to the oracle choice than two alternatives. The first is counting on randomly labeled examples. The second is counting on examples whose labeling was targeted with the same detector. The estimator should also report a validation worst-group number that is not inflated relative to test. The gain comes from two sources. One is the variance argument above. The other is a cancellation of the shared calibration error in paired comparisons between checkpoints. The method's critical assumption is testable: the detector's mistakes must be independent of whether the candidate model is right, given the true group. We expect this assumption to fail exactly when the detector keys on the same spurious feature as the model, and we make that failure a central experiment rather than a footnote.

Positioning

Three findings motivate the proposal. Simple data balancing achieves competitive worst-group-accuracy shows that simple subsampling matches group robust optimization. It also shows that the real dependence on group labels sits in validation-time model selection. That paper treats the validation group labels as given and does not ask how noisy selection on a small minority split is. That noise is our target. On the Limits of Evaluating Embodied Agent Model Generalization Using Validation Sets documents a validation-chosen model falling from 13.8% to 8.57% success on test, while a rival ranked lower on validation reached 15.43%. Stabilizing Minimum Error Rate Training shows that test BLEU varies by up to 2.14 points with the random seed of a development-set search. Both papers are evidence that selection over noisy development sets fails, but neither concerns subgroups.

Two theory papers constrain the design. Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective bounds the train-to-test fairness gap by the inverse harmonic mean of group sizes. Impact of Data Distribution on Fairness Guarantees in Equitable Deep Learning shows fairness error converging at rate $1/\sqrt{m_{\min}}$ in the smallest group size. Both analyze training, not selection. Both imply that any fix must raise the effective minority count, which is what tagging does without new labels.

Semi-supervised Group DRO: Combating Sparsity with Unlabeled Data is the closest relative in spirit, since it uses unlabeled data to compensate for 3 or fewer labeled minority points. It uses that data to train, and it relies on dense-group predictors transferring to sparse groups. We use unlabeled-group data only to measure, and our transfer assumption is a factorization we check directly.

The physics side supplies the method. Absolute Measurement of Hadronic Branching Fractions of the D_s^+ Meson fits 16 single-tag and 64 double-tag yields simultaneously. In doing so it corrects for "cross-feed," where one decay mode is misreconstructed as another, and it lets the production count cancel. We reuse that structure. A tag channel corresponds to a decay mode, detector confusion to cross-feed, and model correctness to the signal decay. We do not reuse anything about the physics itself.

Research Program

Let the validation pool contain $n_g$ examples of true group $g$, with $n_g$ unknown. A detector assigns each example a tag $t$. Let $\epsilon_{tg}$ be the probability that a group-$g$ example receives tag $t$, and $a_g^{(k)}$ the accuracy of candidate model $k$ on group $g$. Class labels are available on validation; only the group attribute is missing. Under factorization, the expected count of examples with tag $t$ and the expected count of those that model $k$ also gets right are

$$\mathbb{E}[N_t]=\sum_g n_g\,\epsilon_{tg},\qquad \mathbb{E}[C^{(k)}_t]=\sum_g n_g\,\epsilon_{tg}\,a^{(k)}_g .$$

We fit $n_g$ and $a_g^{(k)}$ by Poisson likelihood across all tag channels at once. The rates $\epsilon_{tg}$ enter as constrained nuisance parameters estimated from the $B$ hand-labeled calibration examples. The group sizes are fitted, not assumed, so a shift in group prevalence between validation and test does not bias the accuracy estimates. That is the double-tag cancellation.

Where factorization may fail, we add a correction factor $\kappa_{tg}^{(k)}$, defined as the true joint rate divided by $\epsilon_{tg}a_g^{(k)}$. Physics analyses handle the analogous effect with efficiency corrections. We estimate $\kappa$ on the calibration set and report its size as a diagnostic.

A rough calculation shows where the gain comes from. Suppose a detector tags minority examples with purity $p$, and the accuracy difference between majority and minority is $\Delta$. The tag-and-count estimate of minority accuracy has a dominant error of about $\Delta\sqrt{p(1-p)/B}$, which comes from calibrating the purity. Direct counting on roughly $pB$ labeled minority examples has an error of about $\sqrt{a(1-a)/(pB)}$. With $p=0.9$, $\Delta=0.3$, $a=0.7$ and $B=100$, these are roughly 0.009 and 0.048. Both forms follow from binomial variance. For a paired comparison of two checkpoints, the purity error enters only through the difference in their $\Delta$ values, so it shrinks further.

Selection then uses a lower confidence bound on each candidate's fitted worst-group accuracy. The bound is widened for the number of candidates searched, in the way physicists correct a local significance for the "look-elsewhere effect" of scanning many bins. We report the selected model's bound as the validation number.

For the detector, we use zero-shot attribute prompts from a pretrained vision-language model on the image benchmarks. On CivilComments we use a keyword and zero-shot identity detector. These sources are deliberately independent of the candidate models.

Baselines

The naive reference picks the checkpoint with the best average validation accuracy and uses no group labels. Simple data balancing achieves competitive worst-group-accuracy suggests this choice is poor, and it sets the floor.

The second baseline is random-annotation counting. Spend the budget $B$ labeling random validation examples, then select by observed worst-group accuracy on them. This is what a practitioner without group labels would naturally do.

The strongest competitor is detector-targeted annotation. Spend all $B$ labels on examples the detector tags as minority, then count directly on those that are confirmed. It uses the same detector and the same budget as our method, so any win must come from the cross-feed fit and the paired cancellation, not from the detector itself.

The upper reference is full-label selection, which uses true group labels on the entire validation set. This is the protocol of the balancing paper; the file reports no worst-group accuracy number from that paper itself. For calibration, the file's best reported CelebA figure for subsampled training is 15.1% worst-group test error, from Overparameterization hurts worst-group accuracy with spurious correlations. No reported result was found on Lacuna for validation-noise-limited selection on any of these benchmarks.

Three ablations isolate the key ingredients. The first is a plug-in estimator that treats detector tags as true groups, with no cross-feed fit. The second is our fit without the factorization correction. The third uses the fitted point estimate but drops the lower-bound selection rule.

All methods select from one shared pool of candidates. The pool is SUBG, reweighting and ERM, each over a fixed hyperparameter grid and run with five seeds per setting, with checkpoints saved every epoch. Every method sees the same $B$, and every random draw of calibration examples is repeated over 50 resamplings.

Beating a baseline means lower test worst-group regret against the test-oracle checkpoint, with paired confidence intervals over resamplings excluding zero. It also means a smaller gap between the reported validation worst-group number and the actual test number.

Why not just…?

Why not just label more validation examples at random? At realistic budgets, random labeling yields minority counts that are a small fraction of $B$. On CelebA, the hardest group is a few percent of the data. The $1/\sqrt{m}$ limit in the cited fairness bounds then applies to selection directly. Our budget sweep will show where, if ever, random labeling catches up. Below that point, our method is preferable; above it, there is no reason to use it.

Why not just use the detector to choose which examples to label, then count? This is the real threat. If the detector has very high purity and the budget is large, targeted counting may match our method. Our calculation predicts that the advantage grows as purity falls and as the number of candidates rises, because paired cancellation compounds across comparisons. The detector-quality sweep, which degrades purity on purpose, settles this. If targeted counting ties our method across the sweep, the fit adds nothing and we will say so.

Why not just bootstrap or shrink the estimate on the small labeled set? A bootstrap measures the noise but does not reduce it. Shrinking estimates toward each other lowers variance but adds bias, and that bias favors whichever candidates sit near the mean. The lower-confidence-bound ablation, run with bootstrap intervals on random labels, tests whether a better selection rule alone recovers the gain without the extra counts.

Why not just treat detector tags as group labels? This is the plug-in ablation. When accuracy on the majority and minority differs sharply, plug-in tags pull the minority estimate toward majority accuracy. That flatters exactly the models that exploit the shortcut.

Proposed Experiments and Decisive Checks

We run on Waterbirds, CelebA and CivilComments-WILDS, hiding validation group labels except for $B\in\{25,50,100,200\}$. MultiNLI serves as a control. Its spurious attribute, the presence of negation words, is computable exactly, so the detector is perfect and all methods should converge.

The primary metrics are test worst-group regret against the test oracle, rank correlation between estimated and test worst-group accuracy across all candidates, and the validation-minus-test gap of the reported number.

The decisive check is a factorization stress test. We build a second detector from the ERM model's own features, which share its spurious shortcut, and measure how large $\kappa$ becomes and whether the method then degrades below targeted annotation. If selection quality tracks the measured departure from factorization, the assumption is the right diagnostic. If the method fails even with an independent detector, the physics analogy does not transfer and the idea should be dropped.