Valid Statistical Inference After Data Driven Model Selection

This direction is merged into another direction: post-selection inference bias. Its last version stays readable and citable here.

Version 1 of 135 papers42 conceptsfrozen 2026-10-04

In modern data science, we rarely decide on a hypothesis before looking at the data. Instead, we use algorithms like the Lasso, stepwise selection, or clustering to "discover" interesting patterns first. However, using the same data to both select a model and test its significance creates a fundamental statistical trap known as "double dipping" or the "winner’s curse."

As François Bachoc and Cathy Maugis-Rabusseau point out in Selective inference after convex clustering with $\ell_1$ penalization, standard p-values become "anti-conservative" in this scenario. They appear deceptively small, leading researchers to claim "significant" discoveries that are actually just artifacts of the search process.

The Mechanism of Selective Inference

To fix this bias, researchers use a framework called Selective Inference (SI). The core intuition is to adjust our expectations based on the fact that a selection occurred.

Imagine you are testing if two clusters in your data have different means. If you chose those clusters because they looked different, a standard t-test is invalid. Selective inference corrects for this by conditioning on the selection event. Mathematically, this means we only consider the probability distribution of our test statistic within the subset of the data space that would have resulted in the same clusters being chosen.

Jonathan Taylor pioneered this by introducing frameworks to calculate exact p-values for models using groups of variables, such as splines or categorical factors, in Selective inference in regression models with groups of variables.

The Over-Conditioning Trap

While conditioning ensures validity, it introduces a new problem: a massive loss of statistical power.

When we condition a test on the entire execution history of an algorithm—every step, every sign change, and every branching decision—we restrict the sample space so severely that we "throw away" most of the information in the data. This "over-conditioning" results in unnecessarily wide confidence intervals and an inability to detect real effects.

Kazuya Sugiyama, Vo Nguyen Le Duy, and Ichiro Takeuchi argue in More Powerful and General Selective Inference for Stepwise Feature Selection using Homotopy Method that conditioning on the entire history of a feature selection process is the primary reason existing methods are often too conservative to be useful in practice.

Recovering Power through Stability and Homotopy

Recent research focuses on how to maintain statistical validity without paying this "power tax." Three main strategies have emerged:

  1. Algorithmic Stability: Michael Jordan and Tijana Zrnic propose using concepts from differential privacy in Post-selection inference via algorithmic stability. By adding a controlled amount of randomization to the selection process, they create "stable" algorithms where the selection event is less sensitive to small changes in the data, allowing for more powerful inference.
  2. Homotopy Methods: Instead of conditioning on every discrete step of an algorithm, researchers like Diptesh Das use parametric programming to characterize the selection event more broadly. In Fast and More Powerful Selective Inference for Sparse High-Order Interaction Model, they show that by "mining" the path of the algorithm, they can avoid over-conditioning on unnecessary parameters.
  3. Multiscale Bootstrapping: Hidetoshi Shimodaira and Yoshikazu Terada explore using the bootstrap to estimate the selection bias across different scales, providing a more flexible alternative to exact geometric conditioning in Selective inference after feature selection via multiscale bootstrap.

Beyond Simple Regression

The push for valid post-selection inference is expanding into complex unsupervised and transfer learning tasks:

By refining how we condition on our choices, this research direction aims to turn data-driven discovery from a "taboo" practice into a rigorous, high-power statistical tool.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_valid_statistical_inference_after_data_d_84facb5d,
  title = {Valid Statistical Inference After Data Driven Model Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/valid-statistical-inference-after-data-driven-model-selection/rdr_84facb5d009443ea856301936a2039fa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_valid_statistical_inference_after_data_d_84facb5d_v1,
  title = {Valid Statistical Inference After Data Driven Model Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/valid-statistical-inference-after-data-driven-model-selection/rdr_84facb5d009443ea856301936a2039fa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}