Valid Statistical Inference After Data Driven Selection

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2335 papers484 conceptsfrozen 2026-10-04

In modern data science, we rarely start with a single, fixed hypothesis. Instead, we explore: we cluster cells to find new types, we scan brain images for active regions, or we mine massive datasets for interesting patterns. But a fundamental statistical trap awaits this exploratory workflow. If you use the same data to both discover a pattern and test its significance, you are "double-dipping."

This practice, also known as circular analysis, invalidates the very p-values intended to provide scientific rigor. Recent research into Post-Selection Inference (PSI) aims to fix this, allowing researchers to be exploratory without being wrong.

The Intuition of Selection Bias

Imagine you are looking at a field of random noise. If you look closely enough, you will eventually find a small cluster of points that look like a pattern. If you then run a statistical test to ask, "Is this cluster different from the rest of the noise?", the test will almost certainly say "Yes."

The test isn't lying; the cluster is different. However, the test assumes you chose that cluster before looking at the data. Because you chose it specifically because it looked different, the standard "null distribution"—the mathematical yardstick used to measure surprise—is no longer valid. As Jesse M. Zhang and colleagues point out in Towards a Post-clustering Test for Differential Expression, clustering algorithms are designed to maximize separation. They will "force a gap" even in pure noise, leading to a flood of false positives.

Circularity in the Wild

This problem is not theoretical; it has historically plagued high-stakes fields:

  • Neuroimaging: In The Quest for the FFA and Where It Led, researchers reflect on how defining a "functional Region of Interest" (fROI) using the same data used to test it can lead to inflated effect sizes. Nikolaus Kriegeskorte argues in Crossvalidation in Brain Imaging Analysis that this circularity prevents an objective assessment of whether a model actually generalizes.
  • Single-Cell Genomics: Brian Hie notes in Computational Methods for Single-Cell RNA Sequencing that performing differential expression analysis on clusters defined by the same genes leads to "p-value inflation." The test merely validates the criteria used to create the groups in the first place.
  • AI Diagnostics: Even generative AI isn't immune. Rosemary He shows that using the same data to identify an anomaly and test its significance in Alzheimer's diagnosis leads to p-value distributions heavily skewed toward zero, even when no real effect exists.

The Selective Inference Solution

The traditional "gold standard" to avoid double-dipping is data splitting: use half your data to find the pattern and the other half to test it. While safe, this is inefficient, especially when data is scarce.

A more sophisticated approach is Selective Inference (SI). The core idea is to condition the statistical test on the fact that a selection occurred. If we know the "rules" the algorithm used to pick a cluster or a feature, we can adjust our null distribution to account for that "search effort."

Makoto Yamada and Jason D. Lee have pioneered methods like hsicInf and mmdInf (Post Selection Inference with Incomplete Maximum Mean Discrepancy Estimator). These frameworks use kernels to identify non-linear relationships and then apply PSI to ensure the discovered features are truly significant. Instead of a standard Normal distribution, they might use a Truncated Normal distribution, which only considers the range of values that could have been selected by the algorithm.

Beyond Simple Splitting: Data Fission

A recent breakthrough in this area is Data Fission, introduced by James Leiner, Aaditya Ramdas, and Larry A. Wasserman in Data fission: splitting a single data point.

Rather than splitting the dataset (losing half the samples), data fission "splits" each individual data point into two independent components. One component is used for model selection (discovery), and the other for inference (testing). This allows for much higher statistical power while maintaining the rigorous "honesty" required for valid science.

By accounting for the selection process—whether through truncation, knockoffs (Matteo Sesia), or fission—researchers can finally move past the "statistical sin" of double-dipping, turning exploratory data mining into a source of reliable, reproducible discovery.

Papers in this version

Showing 60 of 335 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 266 papers
  2. Version 1 · 2026-10-04 · 335 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_post_selection_inference_bias_c3b58e45,
  title = {post-selection inference bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-selection-inference-bias/rdr_c3b58e4529774e5ab4cc9704f6f1e8d9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_valid_statistical_inference_after_data_d_c3b58e45_v1,
  title = {Valid Statistical Inference After Data Driven Selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/valid-statistical-inference-after-data-driven-selection/rdr_c3b58e4529774e5ab4cc9704f6f1e8d9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}