The Fragility of Fixed Data Splits

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2103 papers144 conceptsfrozen 2026-10-04

In the race to achieve "State of the Art" (SOTA) performance, the machine learning community has largely converged on a single, shared protocol: the fixed train-test split. By evaluating every new model on the exact same slice of data, researchers can compare results down to the second decimal point.

However, a growing body of evidence suggests that these fixed benchmarks are often built on sand. When a dataset is partitioned only once, the resulting performance metrics become highly sensitive to the specific quirks of that partition. What looks like a breakthrough in architectural design is frequently just a "lucky" alignment between a model’s inductive biases and the idiosyncratic distribution of a single test set.

The Mirage of the Standard Split

The core problem is data split bias. In many research areas, such as Natural Language Processing (NLP), the community relies on "standard splits" that have remained unchanged for decades. Kyle Gorman demonstrated the danger of this practice by re-evaluating nine popular part-of-speech taggers across multiple random reshufflings of the same data. He found that system rankings were remarkably unstable; a model that appeared superior on the standard split often underperformed when the data was even slightly perturbed.

This instability isn't limited to NLP. In the field of drug discovery, Dejun Jiang and colleagues found that reported gains for deep graph neural networks were often artifacts of over-optimistic evaluation. When subjected to rigorous cross-validation, traditional "old-school" models like XGBoost often proved more robust and efficient. Similarly, Vera Demberg warned that for tasks with small datasets, like implicit discourse relation classification, a single test set is simply too small to provide a statistically significant signal, leading to "lucky" results that fail to generalize.

Why Rankings Crumble

Why does a model’s rank change when we reshuffle the data? The answer lies in the high variance of performance metrics. John Walker Orr showed that the volatility introduced by random initializations and specific data splits often dwarfs the actual performance gains reported in papers.

This variance creates a "reality check" for the field. For instance, Kuan-Hao Huang introduced TEXTEE, a standardized benchmark for event extraction, to reveal that many "breakthroughs" vanished when models were evaluated across diverse, multi-split partitions. They found that specialized models still vastly outperform Large Language Models (LLMs) once the evaluation is cleaned of data split bias. In morphological tasks, Jordan Kodner found that models struggle significantly more with novel feature combinations than standard splits suggest, because those splits often contain "overlap" that makes the task artificially easy.

The Mechanics of Evaluation Noise

The sensitivity of these metrics is particularly acute in specialized domains:

  • Bio-signals: Eloy Geenjaar noted that in ECG and EMG modeling, the variance caused by different data splits is significantly higher than the variance caused by random seeds, making it nearly impossible to distinguish genuine architectural improvements without rigorous cross-validation.
  • Medical Imaging: Yasmin C. Aguirre found that relying on a single split for skin lesion classification produces metrics that are highly sensitive to the specific composition of the test set, especially given the severe class imbalances in medical data.
  • Social Science: H. de Vos demonstrated that in political research with small samples (N=200), a single split can lead to extreme instability, where 95% of the results might be driven by a statistical fluke.

Toward Robust Validation

To move beyond the fragility of fixed splits, researchers are proposing more resilient validation frameworks.

One approach is Multi-split Cross-Validation, which evaluates models across several distinct partitions to provide a distribution of performance rather than a single point estimate. Abdul Nurunnabi introduced $k$CV-B, a hybrid strategy that combines $k$-fold cross-validation with bootstrapping to generate a "multiplicity" of validation sets, offering more statistically grounded metrics.

An even more radical departure is Mutation Validation (MV), proposed by Jie Zhang. Instead of relying on a hold-out set, MV intentionally corrupts a small portion of training labels and measures the model's reaction. This technique has proven to be more sensitive to overfitting and more stable across different runs than traditional out-of-sample validation.

Ultimately, the goal is to shift the community's focus from "beating the benchmark" to "proving generalizability." As Anders Søgaard argued, the standard statistical thresholds we use are often insufficient; we need stricter validation and a deeper skepticism of results derived from a single, arbitrary slice of data. Only by embracing the variance inherent in our datasets can we build models that truly work in the real world.

Papers in this version

Showing 60 of 103 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 107 papers
  2. Version 1 · 2026-10-04 · 103 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_data_split_sensitivity_05e6ca3e,
  title = {data split sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-split-sensitivity/rdr_05e6ca3e43f04292970e0e1851cf32ca},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_fragility_of_fixed_data_splits_05e6ca3e_v1,
  title = {The Fragility of Fixed Data Splits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-fragility-of-fixed-data-splits/rdr_05e6ca3e43f04292970e0e1851cf32ca/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}