Abstract
For many causal effect parameters of interest, doubly robust machine learning (DRML) estimators $\hatψ_{1}$ are the state-of-the-art, incorporating the good prediction performance of machine learning; the decreased bias of doubly robust estimators; and the analytic tractability and bias reduction of sample splitting with cross fitting. Nonetheless, even in the absence of confounding by unmeasured factors, the nominal $(1 - α)$ Wald confidence interval $\hatψ_{1} \pm z_{α/ 2} \widehat{\mathsf{se}} [\hatψ_{1}]$ may still undercover even in large samples, because the bias of $\hatψ_{1}$ may be of the same or even larger order than its standard error of order $n^{-1/2}$. In this paper, we introduce essentially assumption-free tests that (i) can falsify the null hypothesis that the bias of $\hatψ_{1}$ is of smaller order than its standard error, (ii) can provide an upper confidence bound on the true coverage of the Wald interval, and (iii) are valid under the null under no smoothness/sparsity assumptions on the nuisance parameters. The tests, which we refer to as Assumption Free Empirical Coverage Tests (AFECTs), are based on a U-statistic that estimates part of the bias of $\hatψ_{1}$.
Introduction: The Hidden Bias of Machine Learning
In causal inference, Doubly Robust Machine Learning (DRML) estimators are widely considered the gold standard. By combining two distinct models, they aim to cancel out errors and provide reliable estimates of treatment effects. However, a significant theoretical gap exists: for the resulting nominal $(1 - α)$ Wald confidence intervals to be valid, the estimator's bias must be of a smaller order than its standard error ($n^{-1/2}$).
In high-dimensional "big data" settings, this assumption frequently fails. Hidden biases can be as large as or larger than the standard error, leading to severe "undercoverage." A study might report 95% confidence, while the true coverage is near 0%. Traditionally, testing for such bias required strong, often unverifiable assumptions about data smoothness or sparsity. This paper introduces Assumption-Free Empirical Coverage Tests (AFECTs) to break this circular logic.
Methodology: AFECTs and Second-Order U-Statistics
AFECTs utilize a second-order U-statistic to estimate the "estimable" portion of the bias in $\hatψ_{1}$. Unlike standard estimators that process individual data points, these statistics analyze pairs of points to detect systematic bias. The framework categorizes parameters into two classes:
- Monotone Bias Class: For parameters like expected conditional variance, the test can directly falsify the null hypothesis of negligible bias and provide an upper confidence bound on true coverage.
- General Parameters: For more complex cases, the test acts as a "skeptical" tool, falsifying the analyst's justification for interval validity even when it cannot definitively prove the interval is wrong.
The approach integrates sample splitting and cross-fitting to prevent overfitting and ensure the validity of the tests under the null hypothesis without requiring smoothness assumptions.
Empirical Results
The authors conducted simulations (n=5000) using Hölder smoothness classes designed to induce significant bias. In these scenarios, standard 90% Wald intervals exhibited 0% actual coverage, with bias exceeding the standard error by a factor of 14.
As shown in Figure 3, the AFECT tests successfully rejected the null hypothesis of valid coverage 100% of the time for most values of $k$. Furthermore, the bias-corrected estimator $\hat{\psi}_{2,k}$ demonstrated a clear path to recovery. In the data-adaptive setting (bottom table), as the number of basis functions $k$ increased to 4096, the Monte Carlo coverage improved from 0% to the nominal 90%, while the bias significantly decreased.
Discussion and Future Directions
While AFECTs represent a major step forward, limitations remain. No test can be perfectly consistent against all possible biases without some assumptions; bias hidden beyond the reach of the chosen basis functions will remain undetected. Additionally, while the data-adaptive version performs well in simulations, it currently lacks the full theoretical guarantees of the "oracle" version.
The paper advocates for a "skeptic’s social norm" in statistics: if a result's validity cannot be proven, the analyst has a duty to test and potentially falsify the assumptions underpinning it. Testing for undercoverage should become a standard diagnostic in any machine learning-based causal analysis.
Summary updated