Why Machine Learning Cannot Ignore Maximum Likelihood Estimation

Mark J. van der Laan, Sherri RosearXiv 2021 · arXiv

Introduction: The Inference Gap in Machine Learning

In recent years, machine learning (ML) has undergone an "avalanche" of growth, revolutionizing fields from computer vision to natural language processing. However, this progress has largely focused on predictive accuracy—minimizing error on specific datasets. For ML to transition into a tool for rigorous scientific discovery, it must move beyond "flashy" predictions and provide reliable statistical inference. This paper argues that the path forward requires a return to the foundational principles of Maximum Likelihood Estimation (MLE).

The authors champion the Highly Adaptive LASSO (HAL-MLE) as a powerful "sieve MLE." HAL-MLE bridges the gap between the flexibility of data-adaptive algorithms and the formal requirements of classical statistics, achieving fast, dimension-independent convergence rates and solving the mathematical conditions necessary for valid confidence intervals and hypothesis testing.

The Core Challenge: Score Equations and Uncertainty

Most popular ML algorithms, such as Random Forests or Deep Learning, are designed to optimize prediction. While effective, they often lack the theoretical properties required for formal inference. The core issue is that these models do not inherently solve score equations—mathematical conditions ensuring an estimator is unbiased and efficient enough to allow for the construction of confidence intervals.

Without solving these equations, researchers cannot easily answer critical questions: "How certain are we about this causal effect?" or "Is this result statistically significant?" By framing ML within the context of MLE, the authors provide a framework where uncertainty quantification becomes mathematically grounded rather than heuristic.

Methodology: The Highly Adaptive LASSO (HAL-MLE)

The paper focuses on Sieve Maximum Likelihood Estimation. Rather than fitting a model over an infinite-dimensional space, which inevitably leads to overfitting, a "sieve" uses a sequence of increasingly complex models that grow in tandem with the sample size. The primary tool proposed is the Highly Adaptive LASSO (HAL-MLE), characterized by several distinct features:

  • Indicator Basis Functions: HAL represents complex functions as linear combinations of simple "step functions" (indicator functions) and their interactions. This allows it to approximate virtually any functional form.
  • Sectional Variation Norm: To prevent overfitting, HAL utilizes a complexity measure called the sectional variation norm, enforced via a LASSO ($L_1$) penalty.
  • Score Equation Solving: Unlike many black-box ML methods, HAL-MLE is shown to solve a large class of score equations approximately. This makes it a "plug-in" estimator naturally suited for further statistical refinement.
  • Integration with Targeted Learning (TMLE): HAL serves as an ideal initial step for Targeted Maximum Likelihood Estimation (TMLE). TMLE "nudges" the initial ML fit to specifically solve the score equations for a target parameter (such as a treatment effect), ensuring the final estimate is asymptotically efficient and follows a normal distribution.

Theoretical Advantages: Beyond the Curse of Dimensionality

One of the most significant theoretical contributions of the paper is the "smoothness-free" rate. The authors demonstrate that HAL-MLE achieves a convergence rate of $n^{-1/3}$ (ignoring logarithmic factors) regardless of the number of dimensions, provided the true function has a finite sectional variation norm. This is a remarkable result, as it bypasses the traditional "curse of dimensionality" that typically slows non-parametric methods as the number of variables increases.

Furthermore, because HAL-MLE belongs to a specific mathematical class known as a "Donsker class," the nonparametric bootstrap—a standard method for estimating uncertainty—is theoretically consistent. This is a major advantage over many other ML algorithms where the bootstrap is not guaranteed to provide valid results.

Practical Implications for Causal Inference

In the realm of causal inference, HAL-MLE offers several practical benefits:

  1. Asymptotic Efficiency: When "undersmoothed," HAL-MLE provides efficient estimates for smooth features of a target function, meaning it makes the best possible use of the available data.
  2. Double Robustness: Using HAL to model both the outcome and the treatment/censoring mechanisms (the "nuisance functions") provides a "double robust" layer of protection. If one of the models is slightly misspecified, the final causal estimate can still remain valid.
  3. Interpretability: Despite its complexity, HAL is essentially a high-dimensional linear model using indicator functions. This "closed-form" nature makes it more interpretable and inspectable than many deep learning architectures.

Limitations and Future Directions

While HAL-MLE is theoretically robust, the authors acknowledge practical hurdles. The computational burden is significant; a full HAL model involves $n(2^d - 1)$ basis functions. While strategies like subsetting knots or using "Super Learner" to select complexity make it feasible, more efficient implementations remain an active area of research.

Additionally, while HAL-MLE works for features of a function, proving that the HAL-MLE itself is asymptotically normally distributed at every individual point remains an open mathematical conjecture. The authors also propose a new "score equation preserving TMLE" to ensure that the targeting step does not degrade the existing properties of the HAL model, noting that the evaluation of this approach is a priority for future work.

Conclusion

By returning to the roots of Maximum Likelihood Estimation, the Highly Adaptive LASSO provides a path for machine learning to achieve the statistical rigor required for scientific inference. It offers a unique combination of flexibility, fast convergence, and the mathematical properties necessary to move beyond prediction toward true understanding of data distributions and causal effects.

Summary updated

BibTeX
@misc{laan2021why,
  author = {Mark J. van der Laan and Sherri Rose},
  title = {{Why Machine Learning Cannot Ignore Maximum Likelihood Estimation}},
  year = {2021},
  eprint = {2110.12112},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.12112},
}
Figure