Minimal Conditioning for Powerful Selective Inference

This direction is merged into another direction: post-selection inference bias. Its last version stays readable and citable here.

Version 1 of 119 papers30 conceptsfrozen 2026-10-04

When we use data to both select a model and then test its significance, we fall into the trap of "double dipping." If you search through thousands of genes to find the three most correlated with a disease, standard statistical tests will tell you those three are highly significant—not because they necessarily are, but because you specifically picked the winners. This is selection bias.

Selective Inference (SI) provides a mathematical framework to correct this bias by conditioning the statistical test on the fact that the selection occurred. However, early methods in this field suffered from a major drawback: they were often too conservative, producing p-values so high and confidence intervals so wide that they were practically useless.

Recent research, led by Ichiro Takeuchi and his collaborators, has identified the culprit—over-conditioning—and proposed a solution: minimal conditioning.

The Over-conditioning Trap

To produce a valid p-value after feature selection, we must ask: "What is the probability of seeing this result, given that these specific features were selected?"

Traditional SI methods, such as those based on the "polyhedral lemma," answer this by conditioning on the entire state of the selection algorithm. For a stepwise selection process, this means conditioning not just on the final set of features, but on the exact order they were added, the signs of their coefficients at every step, and every intermediate variable that was rejected.

Mathematically, this restricts the "valid data region" to a tiny, narrow slice of the possible data space. As Kazuya Sugiyama and Vo Nguyen Le Duy demonstrate in More Powerful and General Selective Inference for Stepwise Feature Selection using Homotopy Method, this over-conditioning discards massive amounts of information. When you condition on too many constraints, the resulting distribution becomes highly truncated, leading to a loss of statistical power.

Minimal Conditioning: Less is More

The core insight of this research direction is that we should only condition on the event that is strictly necessary to account for the selection bias.

In Selective inference after feature selection via multiscale bootstrap, Hidetoshi Shimodaira and Yoshikazu Terada argue that conditioning only on the selection of the specific feature of interest—rather than the entire model configuration—prevents the widening of confidence intervals.

By ignoring the "algorithmic history" (like the specific sequence of steps or the signs of coefficients), researchers can define a much larger conditioning event. A larger event means less truncation of the probability distribution, which directly translates to:

  1. Higher Power: A better ability to detect true effects.
  2. Narrower Confidence Intervals: Often reducing the width by half compared to traditional methods.
  3. Actionable Results: Making the findings of complex models, like those for high-order interactions, more reliable for practitioners.

If minimal conditioning is so much better, why wasn't it the standard from the beginning? The challenge is computational.

Conditioning on a simple polyhedral set (the "over-conditioned" approach) is mathematically easy because the boundaries are linear. Minimal conditioning events are much more complex; their boundaries are non-convex and difficult to characterize.

To solve this, Diptesh Das and Ichiro Takeuchi utilize parametric programming and homotopy continuation methods. In Fast and More Powerful Selective Inference for Sparse High-order Interaction Model, they describe a "homotopy mining" framework. Instead of trying to define the entire conditioning region upfront, they move the data vector along a path (a homotopy) and track exactly when the selection outcome changes.

This allows the algorithm to "trace" the boundaries of the minimal conditioning event. By identifying only the pieces of the boundary that the data actually encounters, they can perform exact inference without the computational explosion usually associated with non-convex constraints.

Application to Complex Discovery

This shift toward minimal conditioning is particularly vital in fields like causal discovery and high-order interaction modeling.

In causal discovery, Raanan Y. Rohekar and colleagues show in A Single Iterative Step for Anytime Causal Discovery that prioritizing tests with the smallest possible conditioning sets prevents the erroneous removal of edges in finite datasets. Similarly, when searching for complex multi-mutation drug resistance, the methods developed by Diptesh Das ensure that we can trust the "cherry-picked" interactions without being so conservative that we miss the signal entirely.

By focusing on the final output rather than the algorithmic path, minimal conditioning bridges the gap between the flexibility of modern machine learning and the rigor of classical statistics.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_minimal_conditioning_for_powerful_select_7c66e68c,
  title = {Minimal Conditioning for Powerful Selective Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimal-conditioning-for-powerful-selective-inference/rdr_7c66e68c8a0b400eaea629ee7b0c359d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_minimal_conditioning_for_powerful_select_7c66e68c_v1,
  title = {Minimal Conditioning for Powerful Selective Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimal-conditioning-for-powerful-selective-inference/rdr_7c66e68c8a0b400eaea629ee7b0c359d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}