Imagine a race between two runners. One runner trains for months with a personalized coach, a custom diet, and state-of-the-art shoes tailored to their stride. The other runner is plucked from the stands, handed a pair of generic sneakers, and told to run. If the first runner wins, we cannot logically conclude they are inherently the faster athlete; we have merely demonstrated the value of rigorous preparation.
This experimental imbalance is precisely what happens in machine learning during baseline optimization bias (or hyperparameter optimization bias). When researchers introduce a novel algorithm, they face an asymmetric incentive structure. To prove their method works, they spend a massive portion of their computational budget tuning its hyperparameters to perfection. However, when evaluating the existing baseline models they aim to beat, they rarely afford them the same luxury. Baselines are frequently left with default library settings or configurations copied blindly from older papers.
The result is an illusion of architectural progress, where simpler, classic models appear to be soundly defeated by complex new designs, only because the baselines were never given a fair chance to compete.
Why Uniform Settings Create an Unfair Playing Field
A common attempt to eliminate bias in benchmarking is to apply a uniform training protocol—such as identical learning rates, batch sizes, and weight decays—across all models. While this sounds fair in theory, it is fundamentally flawed in practice.
Architectures of different scales, depths, or mathematical designs possess entirely different optimal training recipes. As Granero Moya et al. demonstrate when evaluating pretrained language models, fixing hyperparameters across models of vastly different sizes inherently disadvantages high-capacity architectures that require specific tuning to unlock their potential. Similarly, Jiang et al. observe that applying a single learning rate across diverse neural network designs fails to reflect the true performance of larger baselines.
Forcing a uniform configuration onto diverse models does not standardize the experiment; instead, it guarantees that the benchmark will favor whichever architecture happens to align best with those arbitrary default settings.
Case Studies in Illusory Progress
When researchers level the playing field by applying an equal hyperparameter optimization budget to both the proposed method and the baselines, supposed architectural breakthroughs frequently evaporate.
Language Modeling
In a foundational study on neural language models, Melis et al. investigated whether complex, newly proposed recurrent cells genuinely outperformed the classic LSTM. By deploying large-scale, black-box hyperparameter optimization to both new architectures and the standard LSTM baseline, the authors revealed that a well-regularized LSTM actually outperformed its newer, more intricate successors. The perceived progress in the field had been an artifact of unequal tuning effort.
Collaborative Filtering
A similar pattern emerged in recommendation systems. Krichene revisited the widely accepted superiority of Neural Collaborative Filtering (NCF) over traditional Matrix Factorization (MF). By applying standard regularization and modern optimization techniques to the classic MF baseline, the author demonstrated that the simpler geometric model consistently outperformed the complex neural architecture. The original claims of NCF's superiority stemmed entirely from an under-tuned baseline.
Apparent Progress (Unequal Tuning):
[Classic Baseline: Default] ──► Low Performance
[New Neural Model: Highly Tuned] ──► High Performance (Claimed Breakthrough)
Real Progress (Equal Tuning Budget):
[Classic Baseline: Highly Tuned] ──► High Performance
[New Neural Model: Highly Tuned] ──► Similar/Lower Performance
Tabular Data
The ongoing debate between Gradient-Boosted Decision Trees (GBDTs) and deep neural networks for tabular data highlights this bias on a massive scale. In an extensive study evaluating 19 algorithms across 176 datasets, Hegde et al. concluded that hyperparameter optimization is often far more critical than model selection. Apparent architectural victories are frequently just reflections of which model received more tuning effort.
The Mechanics of the Bias
Why are machine learning models so sensitive to this bias? Hyperparameters dictate the geometry of a model's loss landscape and the dynamics of how an optimizer navigates it.
When a baseline is left at its default settings, it may suffer from subtle optimization failures—such as getting trapped in suboptimal local minima, experiencing vanishing gradients, or overestimating feature importance. As Gregori et al. note, when standard models are left un-optimized while a proposed method is meticulously refined, the reported performance gains are often entirely illusory, capturing differences in optimization quality rather than inherent algorithmic merit.
Without a rigorous hyperparameter search, it is mathematically impossible to distinguish whether a model performed poorly because its underlying architecture is fundamentally limited, or simply because its learning rate was a factor of ten too high.
Designing Fairer Benchmarks
To move past the illusion of progress, the machine learning community is increasingly adopting benchmarks that standardize the optimization budget rather than the specific hyperparameters themselves.
Instead of comparing a finely tuned model against a default baseline, modern evaluation toolkits like ELEVATER, introduced by Li et al., incorporate automated hyperparameter tuning toolkits directly into the evaluation pipeline. By ensuring that every baseline is automatically optimized for the specific downstream task at hand, researchers can isolate the true value of architectural innovation from the confounding variable of human tuning effort.