Executive Summary
This paper presents a constructive, algorithmic approach to approximating Gateaux derivatives (influence functions) for causal inference using finite differencing. By replacing tedious analytical derivations with a black-box numerical method, practitioners can easily obtain bias-adjusted (e.g., doubly robust) estimators even for complex, constrained problems like infinite-horizon reinforcement learning.
Introduction: The Problem of First-Stage Bias
In causal inference and offline reinforcement learning, researchers often want to estimate statistical functionals, such as the average treatment effect or the value of a specific policy. Simply plugging estimated models (like outcome regressions or transition probabilities) into these functionals often yields poor results due to "first-stage bias" from estimation errors.
To correct this, statisticians traditionally use influence functions—or Gateaux derivatives—to create bias-adjusted, "doubly robust" estimators. However, deriving these influence functions analytically requires significant mathematical effort. If a practitioner wants to tweak an estimand or add custom constraints to a problem (e.g., safety constraints in a Markov Decision Process), they usually have to re-derive the complex math from scratch. This paper asks a practical question: Can we bypass the analytical math and compute these bias adjustments algorithmically using numerical differentiation?
Methodology: Empirical Gateaux Derivatives
The authors propose using empirical Gateaux derivatives computed via finite differencing. Instead of analytically deriving the influence function, the method treats the statistical functional as a black box:
- Plug-in Estimates: It begins with the estimated probability distributions.
- Perturbation: It slightly perturbs these distributions in the direction of a specific data point using a perturbation parameter $\epsilon$. For continuous distributions, it uses a smoothed perturbation (via kernel density estimation) controlled by a bandwidth parameter $\lambda$.
- Finite Differencing: It evaluates the functional on this perturbed distribution and computes the finite difference to approximate the derivative.
While using numerical derivatives for influence functions has been suggested before, this paper's novelty lies in its rigorous characterization of the empirical derivative (using estimated, rather than true, distributions). The authors mathematically map exactly how this finite-difference approach relates to classical analytical estimators (like Augmented Inverse Propensity Weighting, or AIPW) and establish the precise rates at which $\epsilon$ and $\lambda$ must shrink to maintain desirable statistical properties.
Validation on Synthetic Data
As a primarily theoretical and methodological paper, the authors validate their theoretical decompositions using a synthetic 1D dataset ($n=500$) featuring a piecewise linear outcome model and logistic propensity scores. This simple setup effectively visualizes the numerical approximation error across different values of the hyperparameters $\epsilon$ and $\lambda$, and compares the resulting estimators.
Key Insights
- Algorithmic Equivalence: The empirical Gateaux derivative computed via finite differences with kernel density estimates is nearly mathematically equivalent to evaluating the classic doubly-robust estimator (AIPW) using Nadaraya-Watson regression. The only difference is an additive bias introduced by the smoothing kernel.
- Preserving Rate Double Robustness: A major benefit of analytical doubly robust estimators is that they can converge quickly ($O(n^{-1/2})$) even if the underlying machine learning models converge slowly. The authors prove that their numerical approximation preserves this "rate double robustness," provided the perturbation and smoothing parameters are scaled correctly relative to the sample size.
- Flexibility for Complex RL: The true power of this method shines in complex settings like infinite-horizon off-policy evaluation. By formulating the policy value as a linear program, practitioners can use this finite-difference approach to automatically compute bias corrections even when arbitrary constraints (like capping the time an agent spends in a risky state) are added. Doing this analytically would be a massive headache, but the numerical approach handles it seamlessly.
Main Results
- Quantitative Guarantees: The authors prove that if the perturbation $\epsilon$ and smoothing $\lambda$ are chosen such that $\epsilon \lambda^{-d/2}$ and $\lambda^\beta$ shrink faster than specific rates tied to the sample size $n$, the resulting numerical one-step estimator achieves the optimal $O_p(n^{-1/2})$ convergence rate.
- Dynamic Treatment Regimes (DTR): The paper successfully extends the finite-difference decomposition from single-step treatments to multi-stage DTRs, proving that the numerical approach correctly recovers the nested expectations required for sequential bias adjustment.
- Linear Programming OPE: For infinite-horizon Markov Decision Processes, the authors demonstrate that perturbing the constraints and objective of the optimal policy linear program correctly yields the analytical bias correction (the sample Bellman residual) without requiring manual derivation.
Limitations and Future Work
The authors acknowledge a few key limitations. First, to handle continuous distributions, the method requires smoothing, which introduces an unfavorable dependence on dimensionality (the classic "curse of dimensionality"). Second, the theoretical guarantees assume the functional is pathwise differentiable, which may not hold for all non-parametric models and warrants caution.
For future work, the authors suggest developing fully adaptive methods that do not require prior knowledge of the functional's statistical properties to tune the hyperparameters ($\epsilon$ and $\lambda$). They also note that while their work is heavily theoretical, broader empirical comparisons on real-world datasets would be a valuable next step.
Summary updated