Testing Whether Double Machine Learning's Rate Assumption Holds on Real Data

Modern causal estimators such as double machine learning (DML) estimate a treatment effect by first fitting two auxiliary prediction models. One predicts who gets treated (the propensity score). The other predicts the outcome. These auxiliary models are called nuisance functions. DML's confidence intervals are valid only if the errors of the two models, multiplied together, shrink faster than one over the square root of the sample size. Nobody checks this in practice, because on real data the true nuisance functions are unknown. A Monte Carlo study has shown how badly the assumption can fail. When the functions are rough, nominal 90% intervals cover the truth 0% of the time, and the fitted models' prediction loss gives no warning (Assumption-lean falsification tests of rate double-robustness of double-machine-learning estimators). That study, and the test it proposes, used only one-dimensional synthetic functions. So we still do not know two things: whether this failure actually happens on real covariates and real outcome mechanisms, and whether any diagnostic can catch it without ground truth.

We propose a testbed that answers both questions. It is half real and half designed. We start from real randomized trials and use RCT rejection sampling (RCT Rejection Sampling for Causal Estimation Evaluation) to carve out confounded observational subsamples. The real trial gives us the true effect. The outcome mechanism stays fully real, with unknown smoothness. The propensity score is one we design ourselves, so we know it exactly and can make it as rough as we like. This lets us measure DML's actual coverage on real outcome functions while turning a single knob, the roughness of the treatment-selection mechanism. We then ask whether a high-dimensional version of the higher-order falsification test flags the configurations where coverage collapses.

Core Claim

The claim has two parts. First, the rate assumption behind DML can be stress-tested on real data in a way that reveals the true coverage. This works because, in a rejection-sampled RCT, one nuisance function is known exactly and the other is real. Second, a falsification test built on higher-order influence functions can be extended from one-dimensional wavelet bases to learned bases on tabular and text covariates. Such a test should predict real undercoverage better than the checks practitioners use today, namely nuisance prediction error, disagreement between learners, and drift of the estimate as the sample grows.

The mechanism comes from the standard decomposition of the bias of the augmented inverse-propensity-weighted (AIPW, the usual DML form for average effects) estimator. To first order the bias is

$$ B \approx \mathbb{E}\left[\frac{(\hat e(C)-e(C))\,(\hat\mu(C)-\mu(C))}{e(C)}\right] \le \frac{\|\hat e - e\|_2 \, \|\hat \mu - \mu\|_2}{\min_c e(c)}, $$

where $C$ is the covariates, $e$ the true propensity, $\mu$ the true outcome regression, and the hats denote cross-fitted estimates. Intervals are trustworthy only when $B$ is small relative to the standard error, roughly $n^{-1/2}$. In our testbed $e$ is known exactly. That gives three handles. We can compute $\|\hat e - e\|$ exactly. We can estimate $\|\hat\mu - \mu\|$ on the large parent RCT, where the same outcome regression holds. And we can run an oracle-propensity AIPW that has no first-order bias at all. Together these let us attribute any coverage failure to the nuisance-rate bottleneck rather than to bad luck in the draw or to broken identification.

Research Program

Building the testbed. We take a real RCT with randomized treatment $T$, outcome $Y$ and covariates $C$. We accept each unit with a probability set by a designed propensity $e^*(C)$, following the rejection sampler of RCT Rejection Sampling for Causal Estimation Evaluation. That sampler provably preserves $P(C)$ and $P(Y\mid T,C)$, so backdoor adjustment remains valid. The paper reports that it restores 97–98% coverage of 95% intervals, where earlier subsampling schemes gave 0%. This is what makes the testbed trustworthy. The original paper used it only to rank estimators by average-effect bias. We use it instead as a laboratory for nuisance rates.

The roughness knob. The designed propensity takes the form $e^*(C) = \text{sigmoid}(a \cdot g_s(C))$. Here $a$ sets the strength of confounding. The function $g_s$ is a family on a fixed covariate representation (raw features for tabular data, a frozen text embedding for text) whose roughness rises with an index $s$. At low $s$ it is a smooth low-dimensional function. At high $s$ it is a high-frequency function built from compositions of thresholded random projections. This mirrors the controlled Hölder smoothness of the Monte Carlo setups in Assumption-lean falsification tests of rate double-robustness of double-machine-learning estimators. The difference is that the outcome side is real rather than chosen by the researcher. We also cap the propensity at an overlap floor, because weak overlap inflates $B$ through the $1/\min e$ factor. Holding that floor fixed while varying $s$ lets us separate the rate bottleneck from overlap failure.

Ground truth and repeated draws. The reference effect is the difference in means on the full RCT. It carries its own standard error, which we propagate into every coverage computation. We generate repeated observational datasets by bootstrapping the RCT pool and rerunning the rejection sampler. We report two coverage figures: the empirical coverage of DML intervals against the reference, and the distribution of the standardized error, defined as the estimate minus the reference, divided by the combined standard error. The second figure avoids overstating precision when draws share a parent pool.

A falsification test for high-dimensional covariates. The original test estimates a lower bound on the bias term $B$ using third-order and $m$-th order U-statistics. These are averages over triples or $m$-tuples of observations. The test projects onto a $k$-dimensional basis, Daubechies wavelets in one dimension. Our extension replaces the wavelets with data-driven bases: the leading principal components or kernel eigenfunctions of the covariate representation, with $k$ chosen on a held-out fold. Because the test detects only bias that lies inside the span of the basis, it can only lower-bound $B$. The testbed lets us measure how large a share of the true bias the basis captures, since $\hat e - e$ is computable. We also evaluate the paired higher-order bias correction, which the original paper reports raising coverage from 0% to over 80% in its rough setup, on real outcome mechanisms.

Data. We use two RCTs. The first is the Semantic Scholar A/B test (about 70,000 users, with paper titles and abstracts as covariates) used in RCT Rejection Sampling for Causal Estimation Evaluation; it covers the large-sample, high-dimensional text regime. The second is the experimental NSW sample from the LaLonde data (Comparing Experimental and Nonexperimental Methods: What Lessons Have We Learned Four Decades after LaLonde (1986)?); it covers the small-sample tabular regime, where hidden constants and the cost of splitting the sample matter most. Before any of this, we calibrate the test implementation by reproducing the Hölder Monte Carlo results of the falsification paper. The nuisance learners are CatBoost, LightGBM, Lasso and a small neural network, all with five-fold cross-fitting.

One application without ground truth. The LaLonde re-analysis finds that modern estimators fail placebo tests on 1975 pre-treatment earnings even after trimming. It leaves open whether the cause is unobserved confounding or slow nuisance convergence. We will run the validated test on the observational LDW-CPS and LDW-PSID data. If the test does not reject but the placebo still fails, that points to confounding. If it rejects, and higher-order correction moves the placebo estimate toward zero, that implicates the rate bottleneck. Because the test is validated on the testbed first, this attribution is backed by evidence rather than just applying the tool.

Baselines

The do-nothing reference is standard cross-fitted DML/AIPW whose Wald intervals are taken at face value (Double/Debiased/Neyman Machine Learning of Treatment Effects, An Introduction to Double/Debiased Machine Learning). It is the default in applied work. Its expected failure in rough regimes is documented only in simulation: 0% coverage of nominal 90% intervals in Setup I of Assumption-lean falsification tests of rate double-robustness of double-machine-learning estimators. On real data, the closest reported result is that AIPTW with CatBoost performed best on the Semantic Scholar benchmark (RCT Rejection Sampling for Causal Estimation Evaluation). We found no reported numeric bias or coverage for that configuration on Lacuna. We therefore re-run it ourselves on identical draws and treat CatBoost-AIPW as the strongest estimator to beat.

The strongest existing diagnostic is nuisance prediction error under cross-validation. RealCause: Realistic Causal Inference Benchmarking reports that factual outcome RMSE correlates with causal error "often above 80-95% for standardization estimators." That makes it the natural rival for predicting when intervals fail. We feed it the product of the cross-fitted propensity log-loss and the outcome RMSE.

Two cheaper heuristics are also included. The first is learner disagreement, the spread of DML estimates across the four nuisance learners. The second is sample-size drift: rerun DML on random subsamples of size $n/4$, $n/2$ and $n$, and flag cases where the estimate moves by more than its shrinking standard error predicts.

The ablation that isolates our key ingredient runs the original wavelet test on the single covariate direction most correlated with treatment. This separates the value of the learned high-dimensional basis from the value of the higher-order statistic itself. A second ablation, oracle-propensity AIPW, gives the bias-free ceiling.

All diagnostics see the same draws, use the same cross-fitting folds and the same nuisance models, and receive the same budget for tuning hyperparameters. We judge them by the area under the curve (AUC) for predicting a binary label: the configuration is labelled positive when the empirical coverage of a nominal 90% interval falls below 80%. Beating the baselines means two things. Our test's AUC must be clearly above that of prediction error, drift and disagreement, measured across all roughness and overlap settings on both RCTs. And the higher-order corrected intervals must recover at least 80% coverage, the level the original paper reached in simulation, in the configurations the test flags.

Why not just…?

Why not just use flexible learners like CatBoost and trust DML? Gradient boosting may well learn real outcome functions fast enough. If it does, the testbed will show nominal coverage at every roughness level, and that is a publishable negative result in its own right. But the propensity side is under our control. Theory says that pushing its roughness past the point where the product of errors exceeds $n^{-1/2}$ must break coverage whatever learner is used. Whether that point is reached at realistic roughness levels is exactly the empirical question no paper in the file answers.

Why not just select and check models by prediction error? RealCause's correlation holds within an estimator class, and the same paper notes that it breaks down across classes. Prediction loss measures each nuisance error separately. It does not measure their weighted product within the confounded region, and that product is what drives $B$. Still, if prediction error turns out to rank configurations as well as our test does, we will report that the expensive test is unnecessary. The head-to-head AUC comparison settles this directly.

Why not just benchmark on RealCause-style generative simulators? Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption shows that RealCause's IHDP error can reach $6.209 \pm 11.318$ against a true effect of $-0.604$. It also shows that passing distribution-matching tests does not guarantee causal fidelity. A generative simulator also replaces the real outcome mechanism with a neural network's, and that removes the very unknown smoothness we want to probe.

Why not just check whether the estimate drifts with sample size? This is the real threat. It is cheap and needs no basis choice. It is weak when $n$ is small, as in NSW, because estimates on subsamples are noisy. It also cannot tell bias from variance at a single $n$. We include it as a full baseline, and if it matches our test we will recommend it instead.

Decisive Experiments

The first check is calibration: our implementation must reproduce 0% DML coverage and 100% test rejection on Setup I of the falsification paper, and nominal rejection on Setup II. The second check is existence: on the Semantic Scholar testbed, does DML coverage actually fall as the propensity roughness $s$ rises while overlap is held fixed? Oracle-propensity AIPW must stay near nominal throughout, which confirms the failure comes from the nuisance product. The third check is prediction: does the high-dimensional test reach a higher AUC than prediction error, learner disagreement and drift, on both RCTs and at several subsample sizes? We also report the share of the computable bias $B$ that the learned basis captures, since the test can detect only that share. The fourth check is remedy: does higher-order correction restore coverage in the flagged configurations without widening intervals in the unflagged ones? The final step is the LaLonde attribution, which is reported only if the earlier checks pass. Any of these can fail informatively. The most likely surprising outcome is that real outcome regressions are smooth enough that only extreme designed propensities break DML. That would bound the practical scale of the semiparametric bottleneck on real data for the first time.