Replicated Slices as a Validation Metric for Hidden Stratification

A classifier can score 92% overall and still fail on a subgroup nobody labeled. One example is chest X-rays of collapsed lungs where no chest drain is visible yet. The standard fix is to train with a method that protects the worst subgroup. That fix has a quiet dependency. Every such method must choose a checkpoint and hyperparameters, and choosing them well requires a validation set labeled with the very subgroups we do not know about. Remove those labels and good methods collapse. On CelebA, Just Train Twice (JTT) falls to 40.6% worst-group accuracy, per the ablation in Unsupervised Concept Discovery Mitigates Spurious Correlations. Selection by overall accuracy is exactly the averaging that hid the subgroup in the first place.

We propose to replace that choice with a statistic that can be computed without subgroup labels. The procedure first runs automatic failure-slice discovery on one half of the validation set. It then keeps only the slices whose error gap replicates on the other half, beyond what the same discovery procedure finds on shuffled errors. Candidate models are scored by their accuracy on the worst surviving slice. In short, discovered failure slices are hypotheses to be tested, and only the tested ones may drive model selection.

Core Claim

Slice discovery algorithms find groups of inputs where a model underperforms. Examples are GEORGE, which clusters a trained model's own features (No Subclass Left Behind), and Domino, which clusters image-text embeddings jointly with errors (Domino). Both treat the discovered cluster as the end product.

Two lines of evidence say that is unsafe. Where Does My Model Underperform? found that visually coherent slices often do not correspond to failures that reproduce on fresh data. Auditing the Audit found that audit pipelines fail silently. Both papers motivate this proposal. The second also warns that an unverified audit should not be trusted as a selection signal.

Our claim has two parts. The first is that a cross-fitted replication test, calibrated against a permutation null, turns slice discovery into a reliable label-free proxy for worst-group accuracy. The second is that pooling verified slices across all candidate checkpoints is what makes the proxy work for model selection. Pooling matters because a failure mode exposed by one checkpoint becomes a test for every other checkpoint, including ones whose own features would not reveal it.

Unlike CoBalT, the method needs no spatially separable concepts. It therefore applies to text benchmarks such as MultiNLI and CivilComments, where CoBalT's slot-attention concept dictionary cannot be used.

Research Program

The procedure is fixed and the same across all training methods. We randomly split the unlabeled-subgroup validation set into halves A and B.

For each candidate model in the pool (checkpoints × hyperparameters × training methods), we run discovery on A. The discovery step is an error-aware Gaussian mixture in the style of Domino, applied to a fixed embedding so that slice membership means the same thing for every candidate. We try two fixed embeddings, the audited model family's penultimate features and CLIP or a sentence encoder. This directly probes the disagreement between Targeted Data Generation, which favors the model's own representations, and Domino, which favors cross-modal embeddings.

Each discovered slice is then represented by a membership function, the mixture posterior, which can be applied to new points. All slices from all candidates go into one shared library.

On B, we measure each library slice's error gap relative to the full set for the model that produced it. We repeat the whole discovery process on B with that model's correctness labels randomly permuted, which gives a null distribution of the gaps discovery "finds" when no structure exists. A slice is retained if its B-side gap exceeds the null's upper quantile after a Holm correction across the library. We then swap A and B and take the union of retained slices.

Each candidate $m$ is scored by

$$ S(m) = \min_{s \in \mathcal{V}} \mathrm{LCB}_{\alpha}\big(\mathrm{acc}_m(s)\big), $$

where $\mathcal{V}$ is the set of verified slices and $\mathrm{LCB}_{\alpha}$ is a one-sided $1-\alpha$ lower confidence bound on candidate $m$'s accuracy within slice $s$. Using the lower bound penalizes tiny slices whose accuracy is estimated noisily. That noise is the "low slice strength" regime where Domino reports degradation.

Selection picks $\arg\max_m S(m)$.

We will also analyze when verification loses slices that are real. In particular, we will ask whether the required validation size scales with the inverse square of the true gap times slice frequency, which is what a two-sample test of proportions predicts. That analysis gives practitioners a concrete statement of which strata are too rare to audit with the data they have.

Baselines

Overall validation accuracy. This naive reference is what most pipelines do. It is the purest form of aggregate-metric selection and the thing we must beat by a wide margin on every benchmark.

Worst-class validation accuracy. This coarser disaggregation needs only class labels. It is fair because it uses the same validation data and no extra annotation.

Worst-quantile validation loss. This criterion scores a candidate by its mean loss on its highest-loss 10–20% of validation examples, a form of conditional value-at-risk. It is the strongest simple competitor, because it targets the tail without any clustering. We tune the quantile on the same small grid we use for our significance level.

Unverified discovery. This baseline ranks candidates by their worst GEORGE cluster or worst Domino slice, computed on the full validation set without replication. It is the ablation that isolates our key ingredient. It uses identical embeddings, mixture settings, and slice library, differing only in the replication test and the null.

Pooling ablation. Each candidate is scored only on its own verified slices, which isolates the second ingredient.

CoBalT's label-free selection. This is the strongest existing method. Unsupervised Concept Discovery Mitigates Spurious Correlations reports 89.2% worst-group accuracy on CelebA and 89.0–90.6% on Waterbirds without validation group labels, and 80.0% on UrbanCars. Beating it means matching those numbers within noise on images while also working on text, where it does not apply.

Oracle selection. Selection with true group labels is the ceiling. For context, Last Layer Re-Training is Sufficient reports 92.9% on Waterbirds with group labels.

Fairness controls. Every criterion selects from the identical candidate pool. The pool is produced once by fixed training runs of ERM, JTT, subsampling on inferred groups, and last-layer retraining on inferred groups, so compute is equal by construction. Our main metric is selection regret: the oracle-selected model's worst-group accuracy minus the criterion-selected model's. "Beating" a baseline means lower mean regret across seeds and datasets with non-overlapping bootstrap intervals, and in particular closing most of the gap between JTT's collapsed 40.6% and its labeled-selection performance on CelebA.

Why not just…?

Why not just select on worst-quantile loss? It is cheap and label-free, and it is a real threat. Its weakness is that the worst 10% of losses mixes label noise, genuinely ambiguous inputs, and systematic strata, and nothing distinguishes them. We know of no reported result on Lacuna for quantile-loss model selection on these benchmarks. Our experiments settle the question directly with two comparisons. The first is rank correlation with true worst-group accuracy across the candidate pool. The second is a noise-injection condition (random label flips, no spurious structure) where we expect quantile loss to mis-rank candidates and verified slices to be empty. If quantile loss matches our score everywhere, we will report that the clustering adds nothing.

Why not just annotate a few validation examples per group? Sometimes that is the right answer. However, it presupposes knowing the groups. That is the premise CIFA and Coarse race data conceals disparities show is often false, because the failing strata are intersectional or finer than the annotation schema. We will measure the crossover by giving the oracle 10, 25, 50, or 100 labeled examples per true group and reporting the budget at which it overtakes our label-free score.

Why not just use CoBalT's inferred concepts for selection? Where the spurious feature is spatially separable, it is strong, and we treat it as the bar on image benchmarks. Its reliance on slot attention excludes text and entangled global attributes. MultiNLI and CivilComments therefore test whether a modality-agnostic alternative exists at all.

Why not just trust Domino or GEORGE clusters directly? The human study in Where Does My Model Underperform? shows that coherence and correctness come apart. The unverified-discovery ablation measures exactly how much of that gap replication closes.

Proposed Experiments

Main benchmarks. We use Waterbirds, CelebA (blond hair, gender as the hidden attribute), MultiNLI, CivilComments-WILDS, and UrbanCars. Their group labels are withheld from every criterion except the oracle.

Candidate pools come from 3 seeds and a small grid of learning rate, weight decay, and upweighting factor. Checkpoints are taken at every epoch.

Metrics. We report selection regret, Spearman correlation between each criterion and true worst-group accuracy across the pool, and, for the slice library, the precision and recall of verified slices against ground-truth groups using Domino's matching protocol.

Controlled slice strength. A synthetic study injects hidden strata of controlled rarity and gap into CelebA and U-MNIST, tracing where verification begins to miss real slices. A null condition with no injected structure measures the false-verification rate, which should sit near $\alpha$.

Out-of-benchmark check. For a clinical case, we apply the procedure to CXR14 pneumothorax. There Hidden stratification causes clinically meaningful failures reports AUC 0.94 with a chest drain versus 0.77 without. We ask whether a verified slice aligns with the no-drain group, and whether selecting on it narrows that gap.

Decisive outcome. The claim stands if three things hold. On text, our score has clearly higher correlation with true worst-group accuracy than quantile loss. On CelebA, it recovers JTT from its label-free collapse toward labeled-selection performance. Removing either replication or pooling measurably increases regret.