Model Selection Without Validation Holdout Sets

166 concepts135 papers2 proposals

The Validation Tax in Data-Scarce Regimes

In standard machine learning workflows, datasets are routinely partitioned into training, validation, and test sets. The validation set serves as a crucial proxy for unseen data, providing the feedback loop necessary to tune hyperparameters, select architectures, and determine when to stop training.

However, this practice introduces a severe validation data overhead. In data-scarce environments—such as medical diagnostics, rare class classification, or low-budget active learning—every single sample is precious. As noted by Tom J. Viering in Making Learners (More) Monotone, sequestering data for validation directly starves the training process, which can lead to higher overall error and non-monotonic learning behavior where more data paradoxically degrades performance. If a dataset contains only a few dozen samples, setting aside a portion for validation means the model loses a significant fraction of its potential knowledge. Conversely, making the validation set too small makes its feedback signal noisy and unreliable. This dilemma has driven a shared research direction: eliminating or mitigating validation holdout dependency by performing model selection, hyperparameter tuning, and early stopping directly on the training data or via alternative proxies.

In-Sample Risk Estimation and Compression Bounds

How can a model evaluate its own generalization capacity without a holdout set? The key intuition is that a model's interaction with its own training data contains hidden signatures of its ability to generalize. Rather than measuring performance on unseen points, we can mathematically bound or estimate generalization error by analyzing model structure and training dynamics.

One approach is to derive risk metrics directly from the training matrix. The authors of Kernel Alignment Risk Estimator: Risk Prediction from Training Data introduced a framework that predicts the generalization error of kernel ridge regression using exclusively training data. By identifying a "Signal Capture Threshold," this method determines which components of the data the model has successfully mastered versus what is likely noise, allowing hyperparameter optimization without a validation set.

Another perspective treats learning as data compression. Dario Paccagnan et al. developed the Pick-to-Learn (P2L) algorithm in The Pick-to-Learn Algorithm: Empowering Compression for Tight Generalization Bounds and Improved Post-training Performance, which reframes model selection as finding a minimal subset of data that can reconstruct the model's performance. Because P2L computes tight generalization bounds from this compression metric, it eliminates the need for a test or validation split, maximizing post-training performance by keeping all samples in the training loop.

Alternatively, we can design networks to be structurally self-evaluating. Bruno Kacper Mlodozeniec et al. proposed a solution in Hyperparameter Optimization through Neural Network Partitioning by partitioning the neural network itself alongside the data. This creates an internal ecosystem that mimics the Bayesian marginal likelihood, allowing hyperparameter tuning in a single training run without a separate validation pool.

Model-Internal Early Stopping

Early stopping is typically used to halt training when validation loss begins to climb, signaling the onset of overfitting. To eliminate the validation holdout, researchers have found ways to detect overfitting from the internal dynamics of the network or the task structure itself.

In image denoising, Adria Font Calvarons demonstrated in Improved Noise2Noise Denoising With Limited Data that a separate holdout set is unnecessary. By comparing the denoised outputs of identical or highly correlated scenes within the training loop, the proposed Surrogate Noise2Noise (SN2N) framework creates an internal early-stopping criterion that preserves all limited data for training.

At a deeper structural level, Maximilian Weimar et al. explored this via information theory in Fisher Information Flow in Artificial Neural Networks. They tracked how parameter-relevant information flows through network layers during optimization. By monitoring when the network stops absorbing useful structural information and begins memorizing noise, they designed a model-free early stopping criterion that operates entirely without a validation dataset.

Differentiable and Data-Emphasized Objectives

When hyperparameters (like weight decay or dropout rates) are tuned online, algorithms often use a bi-level optimization framework where the inner loop trains weights on training data and the outer loop updates hyperparameters using validation gradients. As Jelena Luketina et al. showed with the $T_1 - T_2$ method in Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters, this requires a dedicated validation set to compute hypergradients, which degrades performance when data is constrained.

To bypass this requirement, researchers have reframed the optimization objective itself to be self-contained. Ethan Harvey et al. introduced the Data-Emphasized Evidence Lower Bound (DE ELBo) in Learning Hyperparameters via a Data-Emphasized Variational Objective. This modified variational objective allows direct gradient-based hyperparameter learning on the full training set, completely removing the validation holdout requirement while slashing tuning time.

Similarly, Yonatan Dukler et al. proposed DIVA (Differentiable Validation) in DIVA: Dataset Derivative of a Learning Task. DIVA treats the dataset itself as a differentiable object, computing the derivative of a learning task with respect to its training samples. This allows the system to prune outliers, select informative samples, and tune data augmentations automatically without ever needing a separate validation set.

Exploiting Noise and Synthetic Proxies

If an explicit evaluation signal is mandatory, how can we generate it without sacrificing real training samples?

One counterintuitive strategy is to intentionally corrupt the training data. In Use of Bad Training Data for Better Predictions, Tal Grossman et al. demonstrated that by systematically scrambling labels or adding noise to the training set, one can observe how a model's stability degrades. This generates a "Noise Sensitivity Signature" that reveals the optimal balance between model complexity and generalization, replacing traditional cross-validation splits.

Alternatively, real data can be preserved entirely for training by offloading the validation role to generative models. Alon Shoshan et al. demonstrated in Synthetic data for model selection that calibrated synthetic datasets can accurately rank and select the best-performing models. This allows practitioners to allocate 100% of their scarce, real-world human-annotated data to the primary training process, bypassing the validation data overhead entirely.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 131 papers
  2. Version 1 · 2026-10-04 · 124 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_validation_data_overhead_db9c31e1,
  title = {validation data overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/validation-data-overhead/rdr_db9c31e1017044e784ccd863dcaea663},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_validation_data_overhead_db9c31e1_v2,
  title = {validation data overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/validation-data-overhead/rdr_db9c31e1017044e784ccd863dcaea663/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake