Probe-Weighted Risk Trees for Finding Dataset Diversity Gaps

Models trained on data collected from a few hospitals, cameras, or robot setups often learn shortcuts: they latch onto who collected the data or how, rather than what the data shows, and they fail when deployed elsewhere. "Collect more diverse data" is the usual advice, but practitioners face a concrete budget question it does not answer. Which of a dozen metadata factors (scanner, site, patient sex, age band, camera angle, background) should the next labeled batch diversify, and does any single ranking of those factors even exist? This proposal borrows an old engineering tool for that question. Hierarchical risk indexing, which ranks many interacting hazards by arranging them in a tree and weighting them through pairwise comparisons, is used here with one key substitution. The comparisons come from small, matched training experiments on the dataset itself, not from experts.

The core idea fits in two sentences. For each pair of metadata factors, we run cheap paired probe trainings that measure how much held-out accuracy drops when the test data shifts on one factor versus the other, and we aggregate these ratios with the standard eigenvector method into a weighted tree that scores the dataset's diversity deficits. The method's consistency check, normally used to reject contradictory expert judgments, becomes a test for factor interactions. When the measured comparisons cannot be put on one scale, the factors compound, and the tree must merge them into a single intersectional node before data acquisition is planned.

Core Claim

We claim that a probe-weighted hierarchical index does two things a flat diversity score cannot. First, it predicts which unseen deployment shift will hurt a model most. Second, it chooses which data to acquire under a fixed labeling budget so that worst-group accuracy improves more than with random acquisition, exhaustive intersectional balancing, or diversity-score maximization, after the probe compute is charged against the method. A secondary claim is that the consistency ratio is a usable detector of intersectional failure, the case where sex and acquisition site, for example, harm a chest X-ray classifier jointly rather than additively.

Where This Sits

Hierarchical risk indexing is built on the analytic hierarchy process (AHP). Experts compare factors two at a time on a ratio scale. The principal eigenvector of the resulting comparison matrix gives the weights, and a consistency ratio (CR) below 0.10 is required. This method is used for cable tunnels (Operation Risk Assessment of Urban Dense Cable Passageway Based on Fuzzy-Analytic Hierarchy Process), fire risk (High-rise Building Group Regional Fire Risk Assessment Model Based on AHP) and multi-perspective tunneling risk (Research on construction risk assessment method of shield tunnel based on subjective and objective weights). The closest machine-learning use is A Novel Artificial General Intelligence Security Evaluation Scheme Based on an Analytic Hierarchy Process Model with a Generic Algorithm. It builds a four-criterion security tree for AI systems and uses a genetic algorithm to push expert matrices below the CR threshold (it reports CR = 0.03131). Its weights still come from people, its datasets are fixed, and it never asks about training-data composition. Our proposal keeps the tree and the eigenvector aggregation, which serve as the supplied method. We replace elicitation with measurement, and we refuse to "repair" inconsistency, because here inconsistency carries information. Enhancing Adversarial Robustness of LLMs with Analytic Hierarchy Process borrows AHP only as a staged prompting pipeline at inference time, so it does not overlap with this question.

On the diversity side, the motivating evidence is Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation. It shows that robot policies learn shortcuts when task-irrelevant factors such as viewpoint differ more across sub-datasets than task-relevant ones. It also shows that targeted augmentation helps: shortcut learning fell by up to 75% and real-world out-of-distribution (OOD) success rose by up to 175%. Its scoring rubric is flat and task-specific, and its warning that uncontrolled diversity can worsen fragmentation is a constraint on our design. Diversity must be directed at the right factor, which is exactly what a ranking supplies. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis shows that a single demographic factor matters on NIH Chest-XRay14 and CheXpert, but it studies one factor at a time. Measuring Dataset Diversity from a Geometric Perspective (PLDiv, a topological diversity score) and the dispersion scores it beats (Vendi Score, DCScore, MagArea) produce one scalar that does not separate harmful from harmless variation. Agent-data work (Beyond Quantity: Trajectory Diversity Scaling for Code Agents; DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use) shows that structured diversity beats volume, but it chooses its diversity axes by hand. Finally, WILDS: A Benchmark of in-the-Wild Distribution Shifts constrains the proposal. Robust training objectives rarely beat plain training there (no meaningful gain in 8 of 10 datasets), which pushes us to intervene on the data rather than the loss.

Research Program

The input is a labeled pool with metadata factors. One example is pooled NIH Chest-XRay14 and CheXpert, with source hospital, sex, age band and view position. We first build a candidate tree. The target is generalization risk. The criteria are acquisition (site, view), population (sex, age) and context. The indicators are individual factors. Each indicator $i$ gets a deficit score $d_i$, defined as coverage shortfall times label coupling. Coverage shortfall is one minus the normalized entropy of the factor's levels. Label coupling is the mutual information between the factor and the label or source, which follows the fragmentation condition from the robot shortcut paper.

Weights come from paired probes. Each probe is a linear head on frozen pretrained features, so hundreds of probes cost less than one full training run. For factors $i$ and $j$, we build a 2×2 design. We hold out one level of $i$ within each stratum of $j$, and vice versa, using matched sample sizes and seeds. The comparison entry is

$$a_{ij} = \frac{H(i \mid j)}{H(j \mid i)},$$

where $H(i \mid j)$ is the average drop in held-out balanced accuracy when the test set shifts on factor $i$ while factor $j$ is held fixed. Because each entry is measured conditionally, the matrix is not consistent by construction. If factors combine additively, then $a_{ij} a_{jk} \approx a_{ik}$ and CR stays low. If two factors compound, the ratios disagree across contexts and CR rises. When CR exceeds 0.10, we merge the pair whose removal most lowers CR into an intersectional node, re-estimate, and repeat. The index score is

$$R = \sum_i w_i d_i,$$

where $w_i$ is the eigenvector weight of indicator $i$, propagated down the tree. Acquisition then spends the label budget on the level combinations that most reduce $R$.

Baselines

The naive reference point is ERM (plain average-loss training) on the current pool plus a randomly drawn acquisition batch of the same size.

The strongest existing data-side competitor is exhaustive intersectional balancing, meaning stratified acquisition that equalizes counts across all factor-level combinations. The balanced-split result in the gender-imbalance paper suggests this works well when there are few factors. On NIH and CheXpert that paper reports no exact AUC figures (no reported result found on Lacuna for numeric chest X-ray balanced-split AUC).

The strongest training-side competitor is Group DRO with groups defined by factor intersections. WILDS reports that it raises CivilComments-WILDS worst-group accuracy from 56.0% to 70.0%, and we will report our CivilComments numbers against that 70.0%.

We also include diversity-score acquisition: greedily maximize Vendi Score or PLDiv of the acquired set.

Three ablations isolate the key ingredient. The direct-harm ablation weights factors by unconditional per-factor harm, with no pairwise design and no tree. The expert-AHP ablation uses clinician-elicited comparisons with genetic-algorithm consistency repair, as in the AGI security scheme. The no-merge ablation keeps the initial tree even when CR is high.

All methods get the same backbone, the same acquisition budget, and the same hyperparameter grid. The probe compute our method uses is given to competitors as extra tuning trials. Beating a baseline means a higher worst-group accuracy (or worst-region correlation on PovertyMap) averaged over five seeds, with non-overlapping 95% bootstrap intervals, with average accuracy no worse than one point below.

Why not just…?

Why not just use a flat diversity score like Vendi or PLDiv? These scores measure how spread out the data is, not which spread matters. PLDiv's own evaluation targets geometric structure, not downstream worst-group failure. The robot shortcut paper shows that variation in task-irrelevant factors can make shortcuts worse when it is coupled to sub-datasets. The diversity-score acquisition baseline tests this directly. If it matches our method, the hierarchy is unnecessary.

Why not just balance every intersection? This is a real threat when there are three or four factors. The number of cells grows multiplicatively, though, and many cells in pooled chest X-ray or FMoW region-by-year data are nearly empty, so balancing spends the budget on rare cells regardless of harm. We will vary the number of factors from three to eight and report where, if anywhere, the two methods cross. If balancing wins at every size, we will say so.

Why not just run Group DRO on intersection groups? WILDS shows that Group DRO helps when groups are known (CivilComments 56.0% to 70.0%), but robust objectives mostly fail to beat ERM elsewhere, and the method needs the group definition chosen in advance. Our index chooses which intersections to treat as groups, so we also test Group DRO run on the merged nodes our tree finds. The two methods may well be complementary rather than rivals.

Why not just measure each factor's harm directly and rank? This is the closest and most serious alternative, and it is the direct-harm ablation. Our bet is that it fails exactly when factors interact, because unconditional harms then misattribute damage. The decisive comparison is therefore on the subset of factor pairs where CR flags interaction. If direct harm is equally good there, the pairwise machinery adds nothing and we drop it.

Proposed Experiments

The primary testbed is pooled NIH Chest-XRay14 and CheXpert, with source hospital, sex, age and view as factors. Secondary testbeds are four WILDS datasets: FMoW-WILDS (region × year), PovertyMap-WILDS (country × urban/rural; ERM drops from 0.60 to 0.45 Pearson correlation in WILDS), CivilComments-WILDS, and Camelyon17-WILDS (hospital; ERM drops from about 93% to about 70%). A robotics check uses LIBERO-Spatial with viewpoint, background and object factors, where the shortcut paper reports near-zero to non-zero OOD success once diversity improves.

The first decisive check is predictive. Hold out one full level, such as an entire hospital or region. Measure the Spearman correlation between the index's predicted harm and the realized drop, compared with flat scores and direct harm.

The second check is prescriptive. Run budgeted acquisition at 5%, 10% and 20% of pool size against all baselines.

The third check calibrates the interaction detector. On a Colored MNIST variant with planted two-factor interactions of known strength, CR should rise monotonically with interaction strength and stay below 0.10 when none is planted. On real data, merged nodes should coincide with intersections that have the largest excess worst-group error.

The proposal fails if, on real benchmarks, CR does not separate interacting from additive pairs, or if direct harm matches the full method on interacting pairs.