When training machine learning models, we face a fundamental rule: never evaluate a model on the same data used to train it. To measure how well a model generalizes, we typically split our data into separate training and validation sets.
However, this standard practice breaks down when data is scarce. If you only have 30 samples, a traditional 80/20 split leaves you with just 24 training examples and a tiny 6-sample validation set. This creates a double-edged dilemma: your model suffers from data starvation during training, and your evaluation metrics become highly volatile and unreliable.
To bypass this constraint, researchers in small-sample regimes turn to an extreme variant of cross-validation: Leave-One-Out Cross-Validation (LOOCV).
The Mechanics of Maximizing Training Data
The core intuition behind LOOCV is simple: if data is precious, we should use as much of it as possible for training in every single validation slice.
For a dataset containing $N$ samples, LOOCV turns validation into an iterative process of $N$ distinct folds. In each fold, the algorithm isolates exactly one sample to act as the test set and trains the model on the remaining $N-1$ samples. The model then makes a prediction on the single withheld sample, and its error is recorded. This cycle repeats $N$ times, ensuring that every data point serves as the validation set exactly once.
Iteration 1: [ Test ] [ Train ] [ Train ] ... [ Train ]
Iteration 2: [ Train ] [ Test ] [ Train ] ... [ Train ]
Iteration 3: [ Train ] [ Train ] [ Test ] ... [ Train ]
...
Iteration N: [ Train ] [ Train ] [ Train ] ... [ Test ]
This exhaustive approach is highly valuable across domains where collecting data is physically or financially constrained. For example, when predicting medical complications from small clinical cohorts, Adeel Abbasi et al. utilized LOOCV in Machine learning to predict hemorrhage and thrombosis during extracorporeal membrane oxygenation to maximize training utility without sacrificing rigorous separation between training and testing data.
We see the exact same strategy deployed when estimating the mechanical properties of structural steel alloys (Tomoko Fukuda et al. in Development of a Neural Network Simulator...) and when mapping soil organic carbon content using sparse remote sensing data (Caiyao Kou et al. in Estimation of Coastal Wetland Soil Organic Carbon Content...). By keeping $N-1$ samples in the training pool, the model evaluated in each fold is virtually identical in size to the final model deployed on the full dataset.
The Bias-Variance Trade-off
Why does this iterative dance work so well, and what are its hidden costs? The answer lies in the classic bias-variance trade-off of statistical estimators.
Low Bias
Standard $k$-fold cross-validation (like 5-fold or 10-fold) underestimates a model's true performance because the model is trained on only 80% or 90% of the available data. This introduces a pessimistic bias. Because LOOCV trains on $N-1$ samples, the training set size is almost perfectly representative of the full dataset. As a result, the performance estimate has nearly zero pessimistic bias.
High Variance
While LOOCV minimizes bias, it often suffers from high variance. In a foundational study, Ron Kohavi explored this phenomenon in A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. Kohavi demonstrated that because the training sets across all $N$ folds are highly overlapping—sharing $N-2$ samples with one another—the resulting models are highly correlated.
If a dataset contains an outlier or a small perturbation, that single anomaly influences almost every single fold's training process. This high correlation between folds means that the overall accuracy estimate can fluctuate wildly if the underlying dataset changes slightly. For this reason, Kohavi's work suggests that stratified 10-fold cross-validation is often preferable for standard model selection tasks, leaving LOOCV primarily for scenarios where data scarcity leaves no other choice.
Domain Variants: The Jackknife and Leave-Pair-Out
Depending on the research community, LOOCV appears under different names and specialized mathematical formulations.
The Jackknife Test
In bioinformatics and structural biology, this strategy is frequently called the Jackknife cross-validation test. When predicting how protein subunits assemble based purely on genetic sequences, Xiao‐Jing Yu et al. used the jackknife test in Classification of protein quaternary structure by functional domain composition to provide an objective, deterministic benchmark that avoids the arbitrary variance introduced by random data partitioning. Similarly, Shuguang Zhao et al. relied on the jackknife test in iATC-mHyb: a hybrid multi-label classifier for predicting the classification of anatomical therapeutic chemicals to guarantee stable, reproducible performance metrics for multi-label drug classification.
Leave-Pair-Out (LPOCV)
Standard LOOCV evaluates a single point at a time, which creates a mathematical roadblock when trying to calculate rank-based metrics like the Area Under the ROC Curve (AUC). You cannot compute an AUC score using a test set of size one.
To solve this, Antti Airola demonstrated in A comparison of AUC estimators in small-sample studies that Leave-Pair-Out Cross-Validation (LPOCV) serves as the optimal alternative. Instead of holding out one sample, LPOCV holds out every possible pair consisting of one positive and one negative example. By evaluating whether the model correctly ranks the positive sample higher than the negative sample across all possible pairs, LPOCV yields a nearly unbiased, low-variance estimate of AUC, even under severe class imbalances.
Application in Modern Few-Shot Learning
The relevance of leave-one-out evaluation extends into modern large language model (LLM) research. In True Few-Shot Learning with Language Models, the authors identified a widespread flaw in how "few-shot" capabilities are reported: researchers frequently use large validation sets to select the best prompts and hyperparameters, quietly violating the premise that data is scarce.
To establish an honest evaluation framework, the authors show that applying leave-one-out cross-validation directly within the tiny few-shot training set allows researchers to select robust prompt configurations without leaking external data. By treating each few-shot example as a held-out target, LOOCV provides a reliable signal for model tuning in ultra-low-data environments.