TL;DR
When estimating complex statistical functionals (like treatment effects), we often rely on "nuisance" models for things like propensity scores. This paper proves that tuning these models for the best prediction accuracy is often a mistake; instead, you must "undersmooth" or "oversmooth" them—and use specific sample-splitting strategies—to achieve the best possible accuracy for your final estimate.
What Problem Does This Paper Address?
In fields like causal inference and conditional independence testing, we are rarely interested in a model's predictions for their own sake. Instead, we want to estimate a functional—a single number like the Average Treatment Effect (ATE). To get there, we usually have to estimate "nuisance functions" first, such as the probability of receiving a treatment (propensity score) or the expected outcome given certain covariates.
The "Double Machine Learning" (DML) framework has become the gold standard for this, using "doubly robust" estimators that remain accurate even if one of the nuisance models is slightly off. However, a major question remains: How should we tune these nuisance models?
Standard practice is to tune them for prediction (e.g., minimizing Mean Squared Error). But when the underlying functions are "rough" (low regularity), prediction-optimal tuning can lead to massive biases in the final functional estimate. This paper investigates the precise interplay between model tuning, sample splitting, and the smoothness of the data to find the recipe for the most accurate final estimate.
Methods
The authors focus on a specific doubly robust functional: the expected conditional covariance, which is a building block for many causal and independence measures. They analyze four types of estimators:
- Integral-based plug-in: Directly integrating the product of nuisance estimates.
- Monte Carlo-based plug-in: Using a subset of data to approximate that integral.
- Newey and Robins plug-in: A simplified version depending on only one nuisance function.
- First-order bias-corrected (Influence Function) estimator: The classic "doubly robust" approach that adds a correction term to a plug-in estimate.
To study these, they use wavelet projection estimators for the nuisance functions. Wavelets are perfect for this theoretical work because they have a "resolution" parameter ($k$) that acts as a dial for smoothing. A higher $k$ means less smoothing (more complexity), while a lower $k$ means more smoothing.
They test these estimators across three data-handling regimes:
- No Sample Splitting: Everything is estimated on the same data.
- Single Sample Splitting: Nuisance functions are estimated on one half, the functional on the other.
- Double Sample Splitting: Each nuisance function and the final functional are estimated on separate, independent slices of data.
Datasets
The paper relies on synthetic numerical simulations to provide "ground truth" for theoretical validation.
- Setup: They generated $n=300$ and $n=30,000$ samples.
- Complexity: They simulated data with three levels of H¨older smoothness ($s \in \{0.05, 0.25, 0.75\}$), representing "low," "medium," and "high" regularity.
- Goal: The simulations were designed to compare "prediction-optimal" resolutions (which minimize the nuisance model's error) against "estimation-optimal" resolutions (which minimize the final functional's error).
Key Insights and Ideas
- The "Prediction-Optimal" Trap: The most important "aha moment" is that tuning a model to be a great predictor often makes it a terrible component for a functional estimator. In low-regularity settings, the bias from a prediction-optimal model doesn't shrink fast enough.
- Undersmoothing is Necessary: To reach the "minimax" (best possible) rate of convergence, you often have to undersmooth. This means choosing a model that is "too complex" for the nuisance function itself (higher variance), because it drastically reduces the bias that would otherwise ruin the final functional estimate.
- Sample Splitting as a Bias Filter: The paper categorizes bias into three types: "own-observation bias," "non-linearity bias," and "approximation bias." They show that double sample splitting is the most powerful tool because it mechanically eliminates the first two, allowing the estimator to succeed even when the data is very rough.
- First-Order Can Be Enough: Previous literature suggested we might need complex "higher-order" bias corrections for rough data. This paper shows that if you use double sample splitting and tune your first-order estimator correctly, you can achieve optimal results without the extra complexity.
Key Results
- Double Sample Splitting Wins: With double sample splitting, the first-order bias-corrected estimator achieves the minimax optimal rate across all smoothness classes.
- The Cost of No Splitting: Without sample splitting, no estimator could achieve the optimal rate in low-regularity regimes. The "own-observation bias" becomes a bottleneck that no amount of tuning can fix.
- Simulation Proof: In the "low regularity" simulation ($n=300$), using the authors' optimal tuning instead of standard prediction-optimal tuning reduced the Mean Squared Error (MSE) of the first-order estimator by a factor of 45.
Limitations and Future Work
- Specific Functional: The results are derived for the expected conditional covariance. While this is a foundational piece of causal inference, the authors note that extending this to the Average Treatment Effect (ATE) is harder because ATE requires estimating the inverse of the propensity score ($1/p(x)$), which introduces new mathematical hurdles.
- Known Density: Much of the primary analysis assumes the density of the covariates ($X$) is known. While they provide an extension for unknown density in the appendix, it adds significant complexity.
- Future Directions: The authors suggest that "Lepski-type methods" could be used in the future to create "adaptive" estimators that automatically find these optimal tuning parameters without the researcher needing to know the exact smoothness of the data beforehand.
Summary updated