Valid Inference Under High Dimensional Regularization Bias

This direction is merged into another direction: shrinkage bias. Its last version stays readable and citable here.

Version 1 of 171 papers73 conceptsfrozen 2026-10-04

In modern data science, we often face the "curse of dimensionality," where the number of potential predictors ($p$) far exceeds the number of observations ($n$). To make sense of this data, researchers rely on regularization—techniques like the Lasso that shrink coefficients toward zero to prevent overfitting.

While regularization is a triumph for prediction, it is a disaster for inference. By intentionally pulling estimates toward zero, these methods introduce a systematic regularization bias. This bias shifts the distribution of the estimator, making classical statistical tools like p-values and confidence intervals fundamentally invalid. A research direction spanning the last decade has developed a rigorous framework to "debias" these models, allowing us to perform valid scientific inference even in high-dimensional settings.

The Shrinkage Trap

To understand the problem, consider the Lasso. It works by adding a penalty to the loss function that encourages sparsity. This is effective because it filters out noise, but it comes at a cost: even the "true" significant variables are pushed toward zero.

As Adel Javanmard and Andrea Montanari explain in Hypothesis Testing in High-Dimensional Regression Under the Gaussian Random Design Model: Asymptotic Theory, this shrinkage means the estimator is no longer centered at the true parameter value. If you try to build a 95% confidence interval around a biased Lasso estimate, that interval will likely miss the truth because it is centered in the wrong place. This bias does not vanish as the sample size increases; it is an inherent artifact of the optimization constraints used to handle high dimensions.

Debiasing the Lasso

The first major breakthrough in this field was the development of the "debiased" or "desparsified" Lasso. The intuition is to treat the initial Lasso estimate as a starting point and then apply a correction term that "undoes" the shrinkage.

In Confidence Intervals and Hypothesis Testing for High-Dimensional Regression, Javanmard and Montanari propose using a surrogate inverse covariance matrix to rotate the residuals back into the original parameter space. This process removes the first-order bias, resulting in an estimator that is approximately Gaussian and centered at the truth. This allows researchers to compute valid p-values for individual variables, even when $p > n$, a feat previously thought to be theoretically impossible at small sample sizes.

From Linear Models to Double Machine Learning

While the debiased Lasso works for linear models, modern causal inference often requires more flexible tools like Random Forests or Neural Networks to account for complex confounding. However, these "black-box" models also suffer from regularization bias and overfitting.

To solve this, Victor Chernozhukov, Whitney K. Newey, and colleagues developed Double Machine Learning (DML). As detailed in An Introduction to Double/Debiased Machine Learning, the framework relies on two key pillars:

  1. Neyman Orthogonality: Researchers construct "orthogonal" moment functions where the target parameter (e.g., a treatment effect) is locally insensitive to errors in the "nuisance" parameters (e.g., the propensity score). This ensures that the bias from the machine learning model doesn't "leak" into the final causal estimate.
  2. Cross-Fitting: The data is split into folds. The nuisance models are trained on one fold and the target effect is estimated on another. This prevents the model from "remembering" specific noise patterns, which Chernozhukov and others show is essential for achieving the standard $\sqrt{n}$ convergence rate.

This approach has been implemented in widely used software libraries like DoubleML, authored by Philipp Bach and others, making robust causal inference accessible to practitioners.

Automating the Cure

One difficulty with DML is that deriving the "orthogonal" correction term for a new statistical problem can be mathematically grueling. Recent work has focused on Automatic Debiased Machine Learning (Auto-DML).

In Automatic Debiased Machine Learning of Causal and Structural Effects, Whitney K. Newey and Rahul Singh introduce a method that uses Riesz representers to automatically learn the necessary debiasing terms. Instead of manually solving for the bias correction, the algorithm learns it directly from the data. This allows the same framework to be applied to a vast range of problems, from heterogeneous treatment effects in Automatic Doubly Robust Forests to complex dynamic models in Inference on weighted average value function in high-dimensional state space.

Why It Matters

The ability to debias high-dimensional models moves machine learning from a "prediction-only" tool to a "discovery" tool. Whether it is identifying significant edges in brain connectivity networks using a debiased multi-task fused lasso or performing online inference on streaming data, the core insight remains the same: we can embrace the power of regularization to handle complexity, provided we have a principled way to subtract the bias it leaves behind.

Papers in this version

Showing 60 of 71 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_valid_inference_under_high_dimensional_r_4de98bbe,
  title = {Valid Inference Under High Dimensional Regularization Bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/valid-inference-under-high-dimensional-regularization-bias/rdr_4de98bbe2b9945ab802265d9652ac3f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_valid_inference_under_high_dimensional_r_4de98bbe_v1,
  title = {Valid Inference Under High Dimensional Regularization Bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/valid-inference-under-high-dimensional-regularization-bias/rdr_4de98bbe2b9945ab802265d9652ac3f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}