Overcoming the Statistical Power Loss of Sample Splitting

221 concepts164 papers2 proposals

In modern statistical learning, researchers often face a fundamental conflict between mathematical validity and data efficiency. To avoid "double-dipping"—using the same data to both select a model and test its significance—the standard prescription is sample splitting. By partitioning a dataset into disjoint sets for discovery and validation, we ensure that the final inference remains unbiased.

However, this safety comes at a steep price. As noted across decades of research, splitting data inherently reduces the effective sample size available for each task, leading to a significant loss of statistical power. This "data tax" is particularly punishing in high-dimensional settings or small-sample regimes where every observation is critical.

The Data Splitting Tradeoff

The core intuition behind sample splitting is the preservation of independence. In two-sample testing, for instance, Gretton et al. explain that one must divide data to optimize kernel parameters on one subset and perform the actual hypothesis test on the other. This prevents the test from becoming overly sensitive to the specific samples used for tuning, which would otherwise inflate Type I error rates.

Yet, this division creates a bottleneck. Clemencon et al. highlight that in "learn-then-test" procedures, the reduction in the final test's sample size can lead to a significant loss of power. Similarly, Lopez-Paz et al. observe that classifier-based tests require large datasets to ensure both effective model training and reliable evaluation; when data is scarce, the split makes it difficult to detect subtle differences between distributions.

This overhead extends to feature selection. Zhao et al. argue that the requirement to split data for evaluation diminishes the power of predictive models to capture complex patterns. In the context of genome-wide association studies, He et al. demonstrate that while stage-wise pipelines manage computational burdens, the resulting loss of samples in validation steps can leave researchers unable to detect genuine, subtle genetic interactions.

The Nuisance of Nuisance Parameters

A major driver of sample splitting is the presence of nuisance components—unknown parts of a model (like propensity scores in medicine) that must be estimated before the target parameter can be analyzed.

Foster et al. and Syrgkanis show that by leveraging Neyman orthogonality, errors in estimating these nuisance components only have a second-order impact on the final model. While this allows for "oracle" convergence rates, the authors acknowledge that the partitioning required for this theoretical guarantee results in significant data inefficiency. In small-sample regimes, the variance introduced by using only a fraction of the data for nuisance estimation can outweigh the benefits of the debiasing itself.

Beyond the Split: Stability and Selection

To mitigate these losses, recent research has focused on methods that allow for "recycling" data or performing inference on the full dataset without compromising rigor.

Algorithmic Stability

One promising alternative is to replace physical data splitting with mathematical constraints on the learning algorithm. Chen et al. prove that sample splitting is unnecessary in debiased machine learning if the estimator is "stable" (e.g., bagged ensembles). By using stable estimators, researchers can utilize the entire dataset for both training and inference, leading to more precise estimates in small-to-moderate samples. Jordan et al. further this idea by using algorithmic stability—derived from differential privacy—to correct for selection bias without the need for expensive data partitions.

Selective Inference

Instead of using a separate set for validation, selective inference allows researchers to perform valid testing on the same data used for discovery by conditioning the test on the selection event. Suzumura et al. demonstrate this in high-order interaction models, identifying significant features while strictly controlling for false positives caused by "double-dipping." This approach recovers the statistical power that would otherwise be lost to a discovery/validation split.

Aggregation and E-values

Other strategies focus on smarter aggregation. Schrab et al. introduce the KSDAGG test, which avoids wasteful splitting for kernel tuning by aggregating multiple tests across a collection of kernels. In a different vein, Pandeva et al. propose using e-values for classifier-based tests. Unlike p-values, e-values allow for "anytime-valid" inference, enabling practitioners to collect data sequentially and stop as soon as a result is clear, rather than being locked into a fixed, inefficient data split.

Conclusion

While sample splitting remains a robust tool for ensuring statistical validity, the "power loss" it incurs is a significant hurdle for modern data science. The transition from rigid partitioning toward methods based on algorithmic stability, selective inference, and adaptive aggregation represents a shift toward a more data-efficient future, where the full weight of available evidence can be brought to bear on every hypothesis.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 175 papers
  2. Version 1 · 2026-10-04 · 197 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_sample_splitting_overhead_dca610d8,
  title = {sample splitting overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-splitting-overhead/rdr_dca610d873984f49b75bc3fd9281de8d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_sample_splitting_overhead_dca610d8_v2,
  title = {sample splitting overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-splitting-overhead/rdr_dca610d873984f49b75bc3fd9281de8d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake