Valid Statistical Inference by Conditioning on Selection Events

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2105 papers162 conceptsfrozen 2026-10-04

In modern data science, we rarely decide which hypothesis to test before looking at the data. Instead, we use algorithms—like the Lasso, stepwise regression, or clustering—to "find" interesting patterns first. This practice, often called "double-dipping," creates a fundamental crisis in statistics: if you use the same data to both discover a feature and test its significance, your p-values will be "anti-conservative," making random noise look like a breakthrough.

Selective Inference (SI) provides a mathematical framework to fix this. By conditioning the statistical test on the fact that a specific model or feature was selected, researchers can produce valid p-values and confidence intervals that account for the "cherry-picking" inherent in the discovery process.

The Geometry of Selection

The core intuition behind selective inference is that an algorithm’s choice restricts the possible values our data could have taken. As Jason D. Lee and Jonathan E. Taylor demonstrated in Exact Post Model Selection Inference for Marginal Screening, many common selection procedures can be viewed as "polyhedral" constraints.

When you run a selection algorithm, you are essentially partitioning the space of all possible datasets into regions. For example, if you use the Lasso to select features, there is a specific "polyhedron" in the data space where that exact set of features would be chosen. If your observed data falls inside that polyhedron, you select those features.

The bias arises because standard statistical tests assume the data could have come from anywhere in the entire space. Selective inference corrects this by conditioning the distribution of the test statistic on the event that the data fell within the selection region.

Inference via Truncated Distributions

Once we identify the selection region, the "honest" way to perform a test is to use a truncated distribution.

In Exact inference after model selection via the Lasso, Dennis L. Sun, Yuekai Sun, and Jonathan Taylor explain that if the original noise in the data was Gaussian, the noise after selection follows a Gaussian distribution truncated to the selection polyhedron.

Instead of comparing an observed effect to a standard bell curve, we compare it to a bell curve that has been "chopped off" at the boundaries of the selection event. This adjustment effectively "subtracts" the information that was already used to pick the model, leaving only the remaining randomness for the significance test. As Jonathan Taylor showed in A significance test for forward stepwise model selection, this ensures that under the null hypothesis, p-values are once again uniformly distributed between 0 and 1.

Beyond Linear Models

While early work focused on linear regression, the framework has rapidly expanded to more complex scenarios:

  • Clustering: When we define groups based on the data and then test if they are different, we are prone to false discoveries. Benjamin Hivert and Boris P. Hejblum in Post-clustering difference testing use SI to condition on the clustering event, ensuring that the groups we "found" are actually statistically distinct.
  • Adaptive Data: In clinical trials or A/B tests where we change our sampling strategy based on early results, the data becomes biased. Xinkun Nie and Jonathan Taylor developed a conditional Maximum Likelihood Estimator to correct this "adaptive bias" by conditioning on the sequence of choices made by the algorithm.
  • Deep Learning and Pipelines: Recent work by Ichiro Takeuchi and Tomohiro Shiraishi has extended these ideas to Vision Transformer attention maps and multi-stage data analysis pipelines. By treating the entire pipeline as a single selection event, they can provide valid p-values for features that survived a complex series of preprocessing and selection steps.

The Advantage over Data Splitting

The traditional way to avoid double-dipping is to split the data: use one half for discovery and the other half for testing. However, this is inefficient and reduces statistical power.

Methods like Data Fission, introduced by James Leiner and Aaditya Ramdas in Data fission: splitting a single data point, offer a more powerful alternative. By "splitting" the information within each data point rather than splitting the observations themselves, researchers can perform valid inference while using the full sample size for both selection and testing.

Ultimately, selective inference allows us to be "exploratory" and "rigorous" at the same time. It acknowledges that we hunt for patterns in data, but provides the mathematical guardrails to ensure that what we find is not just a mirage of the search process itself.

Papers in this version

Showing 60 of 105 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 65 papers
  2. Version 1 · 2026-10-04 · 105 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_selective_inference_83770e6a,
  title = {selective inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-inference/rdr_83770e6a3ab5427aa5e48e59e688a2ac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_valid_statistical_inference_by_condition_83770e6a_v1,
  title = {Valid Statistical Inference by Conditioning on Selection Events},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/valid-statistical-inference-by-conditioning-on-selection-events/rdr_83770e6a3ab5427aa5e48e59e688a2ac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}