Selective Inference for Valid Post-Selection Statistical Testing

This direction is merged into another direction: post-selection inference bias. Its last version stays readable and citable here.

Version 1 of 121 papers21 conceptsfrozen 2026-10-04

In modern data science, we often perform "double-dipping": we search through a dataset to find the most interesting patterns, features, or anomalies, and then we use that same data to test if those discoveries are statistically significant. Standard statistical theory assumes that the hypothesis being tested was chosen before looking at the data. When the hypothesis is instead chosen because of the data—a process known as model selection—traditional p-values become biased, leading to a flood of false positives.

Selective Inference (SI), also known as Post-Selection Inference (PSI), provides a mathematical framework to correct this bias. It allows researchers to use the entire dataset for both discovery and validation without sacrificing statistical rigor.

The Selection Bias Problem

The core issue, as described by Makoto Yamada, Ichiro Takeuchi, and colleagues, is that if you pick the "best" performers out of a hundred random variables, they will appear significant by chance alone. This is the essence of p-hacking.

Traditionally, researchers avoided this by "data splitting"—using half the data to find features and the other half to test them. However, as Jason D. Lee points out, data splitting is inefficient because it reduces the sample size for both stages, significantly lowering the "statistical power" (the ability to detect a true signal). Selective inference offers a way to achieve the same validity as data splitting but with much higher power by utilizing the full dataset.

Conditioning on the Selection Event

The key intuition behind selective inference is to calculate p-values by conditioning on the fact that the selection occurred.

Imagine a high-dimensional space where each point represents a possible dataset. A selection algorithm (like LASSO or a greedy search) partitions this space into regions. If we select a specific feature, it means our dataset fell into a specific "selection region." To get a valid p-value, we don't compare our observed result to all possible outcomes; we only compare it to other possible outcomes that would have resulted in the same selection.

Jason D. Lee and Jonathan Taylor pioneered methods to characterize these selection regions geometrically. For many common algorithms, these regions are "polyhedral." By restricting the probability distribution to this polyhedron, researchers can derive a "truncated" distribution that yields exact, unbiased p-values.

Expanding to Complex Models

While early work focused on linear models like the LASSO, the field has rapidly expanded to more complex discovery mechanisms:

  • Non-linear Relationships: Makoto Yamada and Kenji Fukumizu introduced hsicInf, which uses kernels to perform selective inference on non-linear features and complex outputs.
  • Deep Learning and Graphs: Recent work by Shuichi Nishino and Tomohiro Shiraishi has brought SI to Graph Neural Networks (GNNs), allowing for the statistical validation of "saliency maps" to ensure that identified subgraphs are truly meaningful.
  • Anomaly Detection: Mizuki Niihori and others have applied SI to deep k-nearest neighbor (kNN) and autoencoder-based anomaly detection, providing a way to control false positive rates in high-stakes industrial and medical settings.

Overcoming Practical Hurdles

A major challenge in selective inference is that the "noise level" ($\sigma$) of the data is often unknown. Xiaoying Tian and Jonathan Taylor addressed this by using the square-root LASSO, which allows for simultaneous estimation of model coefficients and noise variance, making SI applicable to real-world high-dimensional datasets where variance is a mystery.

Furthermore, in scenarios like Domain Adaptation, where models transfer knowledge from a source to a target domain, the selection process becomes even more biased. Vo Nguyen Le Duy and Nguyen Thang Loi developed frameworks like SFS-DA and STAND-DA to provide valid p-values for features and anomalies discovered after these complex data transformations.

Why It Works: Power and Control

The recurring theme across this research is the rejection of conservative corrections like the Bonferroni method. While Bonferroni avoids false positives by being extremely strict, it often "throws the baby out with the bathwater" by missing genuine patterns.

By mathematically accounting for the search process—whether it's Shinya Suzumura's pattern mining or Akifumi Yamada's frequency-domain anomaly detection—selective inference maintains a strict bound on the Type I error rate (false discoveries) while preserving the statistical power necessary to identify subtle, real-world signals.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_selective_inference_for_valid_post_selec_54c35d10,
  title = {Selective Inference for Valid Post-Selection Statistical Testing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-inference-for-valid-post-selection-statistical-testing/rdr_54c35d107c4c451ba2b853e845aea3eb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_selective_inference_for_valid_post_selec_54c35d10_v1,
  title = {Selective Inference for Valid Post-Selection Statistical Testing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-inference-for-valid-post-selection-statistical-testing/rdr_54c35d107c4c451ba2b853e845aea3eb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}