In machine learning research, the standard narrative for introducing a new model follows a familiar pattern: an existing baseline architecture is shown to fall short on a benchmark dataset, a novel and more complex architecture is proposed, and empirical results demonstrate a significant boost in performance. This empirical leap is typically attributed to the new model's superior inductive biases or structural innovations.
However, a growing body of rigorous benchmarking literature suggests that much of this apparent progress may be an illusion. The culprit is baseline optimization bias—a systemic evaluation flaw where novel architectures appear superior primarily because they receive extensive hyperparameter optimization (HPO) and modern training infrastructure, while the baselines they are compared against remain severely under-tuned.
The Asymmetric Tuning Trap
To understand why this bias occurs, consider how empirical performance is generated. The final accuracy of a model is not a pure reflection of its architecture $A$; rather, it is a function of both the architecture and its hyperparameter configuration $H$:
$$\text{Performance} = f(A, H)$$
When evaluating a newly proposed model $A_{\text{new}}$ against an established baseline $A_{\text{base}}$, a fair comparison requires finding the optimal hyperparameters for both:
$$\max_{H} f(A_{\text{new}}, H) \quad \text{vs.} \quad \max_{H} f(A_{\text{base}}, H)$$
In practice, however, researchers face asymmetric incentives and compute constraints. Significant effort and budget are poured into searching the hyperparameter space for $A_{\text{new}}$. Conversely, $A_{\text{base}}$ is frequently evaluated using default settings, configurations copied from older literature, or a highly restricted search space. The comparison effectively becomes:
$$f(A_{\text{new}}, H_{\text{optimized}}) \quad \text{vs.} \quad f(A_{\text{base}}, H_{\text{suboptimal}})$$
When the new model wins, the victory is attributed to the transition from $A_{\text{base}}$ to $A_{\text{new}}$, masking the fact that the performance gain was actually driven by the transition from $H_{\text{suboptimal}}$ to $H_{\text{optimized}}$.
Why Hyperparameters Outweigh Architecture
Why does hyperparameter selection exert such a dominant influence on final performance, often eclipsing structural design?
Deep learning models operate on highly complex, non-convex loss landscapes. An architectural innovation—such as adding an attention mechanism or a specialized message-passing layer—is intended to reshape this landscape to make desirable minima more accessible. However, optimization hyperparameters (such as learning rates, batch sizes, and learning rate schedules) and regularization hyperparameters (such as dropout rates and weight decay) dictate how effectively an optimizer can navigate any given landscape.
If a baseline model is configured with an suboptimal learning rate or inadequate regularization, the optimizer may get trapped in a poor local minimum or overfit early. This failure is fundamentally an optimization failure, not an architectural one. As Brockschmidt observes in GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation, without a unified and rigorous search space for all compared models, it is impossible to isolate whether performance gains stem from structural innovations or simply superior tuning.
Furthermore, "modern training tricks"—such as layer normalization, residual connections, and advanced weight initializations—are often bundled into new architectures but omitted from baselines. These components smoothen the loss landscape, allowing for more stable training and higher learning rates. When classic architectures are retrofitted with these exact same optimization aids, the perceived superiority of newer, more complex models frequently evaporates.
Empirical Evidence Across Domains
This optimization bias is not isolated to a single subfield; it has been systematically exposed across a wide range of machine learning domains.
Graph Representation Learning
The field of Graph Neural Networks (GNNs) has been particularly prone to this pitfall. Early warnings from Shchur et al. in Pitfalls of Graph Neural Network Evaluation highlighted that the impact of training routines and hyperparameter selection often outweighs the influence of the underlying model architecture on final accuracy.
When rigorous, uniform optimization is applied, leaderboards undergo dramatic shifts:
- Classic GNNs vs. Transformers: It has been widely claimed that modern Graph Transformers have rendered classic GNNs obsolete. However, Luo et al. demonstrate in Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification that when classic models like GCN and GAT are updated with standard modern training tricks (like normalization and residual connections) and properly tuned, they actually outperform state-of-the-art Transformers on 17 out of 18 major benchmarks.
- Link Prediction and Heterophily: In link prediction tasks, Shomer et al. show in Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking that foundational GNN architectures match or exceed complex modern models when properly optimized. Similarly, in heterophilic graph learning, Luan et al. reveal in Re-evaluating the Advancements of Heterophilic Graph Learning that simple Multi-Layer Perceptrons (MLPs) and standard GCNs frequently outperform highly specialized heterophilic models when given equal optimization effort.
- Task-Specific Benchmarking: In applications like materials chemistry, Fung et al. establish in Benchmarking graph neural networks for materials chemistry that systematic HPO significantly reduces the performance gap between various state-of-the-art GNN architectures, proving that the training configuration is as critical as the model structure itself. Even in semi-supervised learning, the authors of Rethinking the Setting of Semi-supervised Learning on Graphs demonstrate that aggressive hyperparameter tuning enables basic graph convolutional networks to match or exceed complex state-of-the-art architectures.
Knowledge Graph Embeddings
A parallel reality check occurred in the evaluation of Knowledge Graph Embedding Models (KGEMs). Ali et al., in their large-scale evaluation Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework, revealed that many "state-of-the-art" gains were driven by training tricks and hyperparameter tuning rather than model architecture. This is reinforced by Alam et al. in Towards Semantically Enriched Embeddings for Knowledge Graph Completion, who note that simple embedding models often match or exceed the performance of complex modern architectures when hyperparameters are properly optimized.
Natural Language Processing and Computer Vision
The illusion of progress via complexity extends to core NLP and vision tasks:
- Semantic Dependency Parsing: Manning et al. show in Simpler but More Accurate Semantic Dependency Parsing that optimizing basic hyperparameters, such as hidden layer dimensions and dropout rates, yields greater performance improvements than introducing complex, specialized decoding algorithms.
- Medical Code Prediction: Liu et al. demonstrate in Parameter Selection: Why We Should Pay More Attention to It that simply re-tuning the hyperparameters of a standard 2018 baseline model allows it to outperform many subsequent "state-of-the-art" architectures.
- Small-Dataset Image Classification: In data-scarce regimes, Denzler et al. show in Image Classification with Small Datasets: Overview and Benchmark that thorough hyperparameter optimization of standard baseline models often yields performance gains that exceed those of specialized architectural or loss-function modifications.
Traditional Heuristics
The bias even distorts comparisons between deep learning and traditional algorithms. Skitsas et al., in Comprehensive Evaluation of Algorithms for Unrestricted Graph Alignment, found that classic structural alignment algorithms remain highly competitive with modern deep learning approaches when their hyperparameters are properly tuned for network size and combined with modern assignment solvers.
The Theoretical Blindspot
If optimization parameters are so critical to empirical success, why does the literature consistently over-emphasize architectural design? Part of the issue stems from a disconnect between learning theory and empirical practice.
As argued by Du et al. in Tuning Algorithmic and Architectural Hyperparameters in Graph-Based Semi-Supervised Learning with Provable Guarantees, current theoretical frameworks for graph-based learning focus primarily on bounding the complexity of architectural and algorithmic hyperparameters. Crucial optimization-specific variables—such as learning rates and batch sizes—are frequently excluded from these theoretical bounds, despite their massive empirical impact on model performance and convergence. This theoretical exclusion inadvertently legitimizes the empirical neglect of baseline optimization.
Restoring Rigor to Machine Learning Evaluation
The collective evidence from these studies provides a clear roadmap for moving past the illusion of architectural progress. To establish true algorithmic advancement, the community must adopt evaluation protocols that treat optimization as a controlled variable rather than an afterthought.
- Unified Search Spaces: Comparisons must occur within a unified framework where baselines and new models are granted the same budget and scope for hyperparameter optimization.
- Ablation of Training Tricks: Performance gains must be explicitly decoupled from optimization enhancements like residual connections, normalization layers, and advanced learning rate schedulers.
- Sober Baseline Re-evaluation: Before proposing a more complex model, researchers should exhaustively tune existing simpler baselines. Often, the path to higher accuracy does not require a more intricate architecture, but a deeper understanding of the optimization landscape we already possess.
Key Foundational Work
GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation (2020) — contributed 3 concept elements to this direction.
Pitfalls of Graph Neural Network Evaluation (2018) — contributed 1 concept elements to this direction.
Simpler but More Accurate Semantic Dependency Parsing (2018) — contributed 1 concept elements to this direction.