In the pursuit of state-of-the-art (SOTA) performance, machine learning research has trended toward increasingly intricate architectures. We often assume that complex problems—like multi-agent coordination, 3D medical segmentation, or personalized recommendation—require equally complex solutions. However, a growing body of evidence suggests that this "complexity bias" may be creating an illusion of progress. When simple, foundational models are tuned with the same rigor as their sophisticated counterparts, the performance gap often vanishes.
The Illusion of Architectural Superiority
The primary driver of perceived progress in many fields is not necessarily architectural innovation, but rather the discrepancy in hyperparameter optimization (HPO) between new models and old baselines. As Desheng Cai and colleagues demonstrate in GRecX, many reported gains in Graph Neural Network (GNN) recommendation are erased when classic Matrix Factorization is properly tuned.
This phenomenon, often called "baseline parity," suggests that we are frequently comparing "well-tuned new models" against "under-optimized old models." Jie-Jyun Liu and Chih-Jen Lin illustrated this by re-tuning a standard 2018 baseline for medical code prediction, finding that it outperformed years of subsequent "SOTA" architectures Parameter Selection: Why We Should Pay More Attention to It.
Why Simple Models Keep Up
The intuition behind why simple models remain competitive lies in the convergence of performance. As Cynthia Rudin and Ronald Parr argue, for many real-world datasets, there exists a large set of near-optimal models (the "Rashomon set"). If multiple disparate architectures yield similar accuracy, it is highly probable that a simpler, more interpretable model exists within that set On the Existence of Simpler Machine Learning Models.
Furthermore, complex models often overfit to the specific quirks of a benchmark. In the realm of 3D medical segmentation, Karol Gotkowski and Paul F Jaeger found that specialized algorithms for sparse supervision failed to generalize beyond their original datasets. In contrast, an "embarrassingly simple" baseline—a standard nnU-Net with a partial loss—consistently delivered superior results across diverse clinical scenarios Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation.
Complexity as a Generalization Barrier
In many domains, the "sophisticated" mechanisms we add to models can actually act as noise or hinder optimization:
- Reinforcement Learning: In multi-agent settings, researchers often favor complex, domain-specific architectures. However, Chao Yu and Yi Wu showed that a well-implemented version of the standard Proximal Policy Optimization (PPO) algorithm matches or beats specialized off-policy methods The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games.
- Transformer Modifications: A massive study by Yi Tay at Google evaluated dozens of proposed Transformer tweaks, finding that most failed to provide consistent benefits over the original, simpler architecture when tested at scale Do Transformer Modifications Transfer Across Implementations and Applications?.
- Optimization: Even in industrial 2D packing, Stefan Böhm found that a simple "Best Fit" heuristic outperformed deep reinforcement learning in both speed and feasibility Comparing Heuristics, Constraint Optimization, and Reinforcement Learning for an Industrial 2D Packing Problem.
The Path Toward Rigorous Benchmarking
To combat the trend of unnecessary complexity, researchers are advocating for "Negative Results" and standardized benchmarking frameworks. Initiatives like BARS Jieming Zhu and pyKT Zitao Liu provide unified platforms that force new models to compete on a level playing field against optimized baselines.
The goal is not to discourage innovation, but to ensure that complexity is earned. As the Proceedings of the Second Workshop on Insights from Negative Results in NLP suggests, understanding why a simple model succeeds is often more valuable than building a complex one that works for reasons we don't fully understand. By prioritizing the "performance floor" established by simple models, the field can move away from marginal, implementation-specific gains and toward genuine architectural breakthroughs.