Benchmark audits regularly find "shortcuts": surface features, such as word overlap between question and passage, that predict the answer without any real reasoning. Today these audits are run with ad hoc classifiers and eyeballed correlations. Nothing bounds how often an audit wrongly accuses a dataset of containing a shortcut. That gap matters when hundreds of candidate surface features are screened against only a few hundred annotated examples, which is exactly how such audits are run. We propose an auditor built on the graphical lasso, a method that estimates a sparse network of direct dependencies among variables. We tune it so that a user-chosen number $\alpha$ bounds the probability of reporting even one spurious shortcut.
In plain terms, the auditor treats each candidate surface feature and the label as nodes in a dependency network. It estimates which features connect to the label directly rather than through some other feature, and it picks its single tuning knob from a desired error level instead of by cross-validation. The output is a short list of shortcuts paired with an explicit statement: with probability at least $1-\alpha$, no feature on this list is a false alarm. The new step is not the estimator. It is moving error-controlled graph selection into data auditing, and testing honestly whether its guarantee survives the discrete, small-sample, high-dimensional conditions of real audits.
Core Claim
Family-wise error rate control in Gaussian graphical model selection via distributionally robust optimization shows that the graphical lasso penalty can be derived from a worst-case-risk ("distributionally robust") problem. Its confidence level $\alpha$ then bounds the asymptotic family-wise error rate, the probability of selecting at least one false edge. The resulting procedure, RobSel, picks the penalty with a bootstrap and was validated only on gene-network data (DREAM5).
Our claim is twofold. First, the edges touching the label node of a RobSel-tuned graph give a shortcut audit with a usable false-alarm guarantee. Second, this audit separates direct label dependencies from indirect ones. For example, it can tell whether answer length predicts the label only because it tracks lexical overlap.
The guarantee is formally about linear partial correlation under Gaussianity. Audit features are counts and indicators, and the label is discrete. So the central empirical question is whether the bootstrap-calibrated $\alpha$ still holds in that setting. That question is itself an open problem the RobSel paper flags (finite-sample calibration away from Gaussianity). We expect the method to add value mainly where the number of candidate features $d$ is comparable to or larger than the number of audited examples $n$. If $n \gg d$, classical tests should suffice, and our experiments are built to show where that boundary lies.
Research Program
The prior work plays three distinct roles here.
Two papers motivate the problem. A Framework for Evaluation of Machine Reading Comprehension Gold Standards audited six reading-comprehension datasets of 300 samples each. It found, for instance, that a word-overlap baseline selects DROP supporting facts at 0.66 F1, but it used shallow classifiers with no error control and no way to separate direct from indirect cues. Diversity, Density, and Homogeneity shows that cheap, training-free dataset statistics predict downstream accuracy. However, its geometric summaries say nothing about which specific features are entangled with labels.
Two papers supply the method. RobSel provides the penalty-from-$\alpha$ mapping. The step-down max-T testing of Perlman (2007) is the classical way to control family-wise error on partial correlations, and it serves as our main competitor.
One paper constrains the design. Robust Gaussian Graphical Modeling with the Trimmed Graphical Lasso warns that a few contaminated rows distort covariance estimates. We therefore test a trimmed variant of the auditor.
The pipeline has four steps. For each dataset, we extract a fixed library of a few hundred candidate surface features: lexical overlap at several granularities, question-word type, answer position, answer and passage lengths, template n-gram indicators, and entity-type indicators. We rank-transform all features and the label. We then fit the graphical lasso at the RobSel-selected penalty for each target $\alpha$ and report the label node's neighbors.
The analysis has three parts. We measure empirical family-wise error rate and power as functions of $d/n$, of feature discreteness, and of label cardinality. We characterize when the bootstrap penalty is miscalibrated. Where it is miscalibrated, we test a simple repair: a permutation-of-labels recalibration of the penalty, which is exact under the null of no label dependence, at the cost of extra compute.
Baselines
The naive reference point is the common practice of testing each feature's marginal correlation with the label at the 0.05 level, with no multiplicity correction. We report how many shortcuts it declares and, on planted-artifact data where truth is known, how many of those are false. Beating it means a far lower false-alarm rate at modest power loss.
The audit-practice baseline is the shallow-classifier approach of the reading-comprehension gold-standards paper: an L1-penalized logistic regression on the same feature library, with its penalty chosen by cross-validation, flagging nonzero coefficients. Its reported strength is the DROP word-overlap result of 0.66 F1. That number measures how predictive the shortcut is, not how accurate the audit is, and no reported false-discovery figure exists for it. The fair comparison runs it on identical features and samples with an equal tuning budget. Beating it means matching its recall of planted shortcuts while keeping family-wise error near $\alpha$, where we expect its error to sit far above $\alpha$.
The strongest statistical competitor is step-down max-T testing of partial correlations following Perlman (2007). It is feasible whenever $n > d$ and is exact in the Gaussian case. We run it at the same $\alpha$, with max-T resampling given the same number of resamples as RobSel's bootstrap. No reported result was found on Lacuna for its error rate or power on audit-style discrete features. The RobSel paper reports only qualitatively that RobSel trades lower power at small $\alpha$ for higher power at larger $\alpha$ relative to Holm testing. Beating this baseline means higher power at matched empirical error when $d/n$ approaches or exceeds one. If Perlman matches us everywhere with $n \gg d$, we report that plainly.
We also run the graphical lasso tuned by cross-validation and by extended BIC, the selectors RobSel was compared against on DREAM5, where cross-validation chose dense graphs. This comparison isolates the value of choosing the penalty from $\alpha$.
Two ablations isolate our key ingredients. The first replaces the conditional graph with marginal-correlation screening at the same $\alpha$ using Holm correction. It tests whether the direct-versus-indirect distinction matters. The second compares asymptotic, bootstrap, and permutation penalty choices.
Why not just…?
Why not just run Holm or max-T tests on partial correlations? This is the real threat. When $n$ is much larger than $d$, partial correlations from ordinary regression are well estimated and step-down testing (Perlman, 2007) controls error exactly under Gaussianity, so the graphical lasso may add nothing. The proposal does not argue around this. The $d/n$ sweep will locate where classical testing breaks down: partial correlations become unstable as $d$ nears $n$ and undefined beyond it. If no such regime exists for realistic audit sizes, the contribution shrinks to the calibration study. The 300-sample reading-comprehension audits suggest the regime is real.
Why not just use lasso logistic regression, which is what auditors already do? Cross-validation tunes for prediction, not for false alarms. RobSel's own comparison found that cross-validated graph selection admits many false edges. The planted-artifact experiment will measure this directly for the classifier approach, by counting how often it flags null features that were planted.
Why not just control the false discovery rate with Benjamini–Hochberg instead? False-discovery control, as in Strobl et al. (2019) for causal graphs, is more powerful and is a reasonable alternative for exploratory audits. We choose family-wise control because the use case is accusation: a published claim that a benchmark has a shortcut. There, a single false claim is costly. We nonetheless report a Benjamini–Yekutieli variant over edge p-values so that readers can see the power gap.
Proposed Experiments
The decisive experiment uses planted artifacts on SST-2, Snips, and the six reading-comprehension sets (MSMARCO, HotpotQA, ReCoRd, MultiRC, NewsQA, DROP). We inject $k$ synthetic tokens whose presence is tied to the label at controlled strengths. Some are tied directly. Others are tied only through an existing feature, which creates indirect cues the auditor should not flag. We also inject many label-independent null tokens. Sample sizes range from 100 to 5,000 examples and feature counts from 50 to 1,000.
Metrics are empirical family-wise error over 500 replicates, power to detect direct planted cues, and the rate of wrongly flagged indirect cues. The proposal succeeds if family-wise error stays at or below $\alpha$ across these conditions, with or without permutation recalibration, and power exceeds max-T testing once $d/n$ passes roughly 0.5. It fails if the bootstrap is badly anti-conservative on discrete features and the permutation repair does not fix it.
The second experiment applies the calibrated auditor to the unmodified reading-comprehension datasets. It checks whether the auditor recovers the word-overlap dependence behind DROP's 0.66 F1 as a direct edge, and whether it attributes NewsQA's near-total retrieval-style questions to a small set of direct cues.
The third experiment tests usefulness. On SST-2, we train the same classifier on the original data and on data rebalanced to break each flagged shortcut, holding the architecture fixed in the DataPerf style. We then compare accuracy on inputs where the flagged feature is decoupled from the label. If rebalancing flagged features helps more than rebalancing an equal number of features flagged by the logistic baseline, the guarantee has bought something practical.
References
- Family‐wise error rate control in Gaussian graphical model selection via distributionally robust optimization
- A Framework for Evaluation of Machine Reading Comprehension Gold Standards
- Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections
- Multiple Testing and Error Control in Gaussian Graphical Model Selection
- Robust Gaussian Graphical Modeling with the Trimmed Graphical Lasso
- Estimating and Controlling the False Discovery Rate of the PC Algorithm Using Edge-specific P-Values