Many papers report a crossover between a simple model and a more elaborate one. The elaborate model, with more built-in assumptions about the data, wins when data is scarce. The simple model catches up or pulls ahead as data grows. These claims rest on "scaling curves": error plotted against training-set size and fitted with a power law. Almost all such curves are drawn with each architecture's hyperparameters fixed across scales, or tuned once and reused. A separate body of work shows that under-tuning alone can create apparent architectural gaps. A re-tuned SECOND detector rose from 51.89 to 63.22 mAP on ONCE with no new data (Refining the ONCE Benchmark With Hyperparameter Tuning). Under equal tuning, the oldest graph network, GCN, came out best (Pitfalls of Graph Neural Network Evaluation). Optimization theory also predicts that the best learning rate, momentum and batch size drift systematically as the training budget grows (Deriving Hyperparameter Scaling Laws via Modern Optimization Theory). Put these together and an uncomfortable possibility follows: the slopes of published scaling curves, and the data size at which a simple model "overtakes" a complex one, may partly measure which architecture tolerates stale hyperparameters, not which one uses data better.
This note proposes to measure that bias directly. We redraw simple-versus-complex scaling curves in two domains under four tuning protocols with matched search budgets. We then report how much the fitted slope and the crossover point move when each model is tuned at each data size. The two domains are molecular property prediction on subsets of QM1B and image pretraining on subsets of ImageNet-21k. The deliverable is a correction, with error bars, to a kind of claim that the field currently makes without one.
Core Claim
Write each model's held-out error as a function of training-set size $D$ as $L(D) = E + B D^{-\alpha}$, where $E$ is the error floor, $B$ a scale constant and $\alpha$ the scaling exponent. For a simple model $s$ and a complex model $c$ with $\alpha_s > \alpha_c$, the curves cross at
$$ D^* = \left(\frac{B_s}{B_c}\right)^{1/(\alpha_s - \alpha_c)}, $$
ignoring the floors. Because $D^*$ depends on the exponent difference raised to a reciprocal power, small biases in either exponent move the crossover by orders of magnitude.
Our hypothesis is twofold. First, fixed or transferred hyperparameters bias $\alpha$ by different amounts for different architectures. Second, the bias is largest for the models whose optimization is most fragile, which in the literature are often the "efficient" or heavily engineered ones. We will define the tuning-induced exponent bias $\Delta\alpha = \alpha_{\text{per-scale}} - \alpha_{\text{fixed}}$ for each architecture and report it, along with the shift in $D^*$, with bootstrap confidence intervals. A null result, where $\Delta\alpha$ falls inside its interval for all architectures, is also a clean and useful outcome. It would license the fixed-hyperparameter practice of Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?.
Positioning
Two groups of prior work motivate the study. The first is the scaling-under-inductive-bias work. Scaling Laws vs Model Architectures fits slopes for more than ten architectures on C4 and finds the vanilla Transformer steepest ($\alpha = 0.54$) and Performer flat ($\alpha = 0.25$). It uses a shared training recipe and does not re-tune per scale; we test whether such slope gaps survive per-scale tuning. Scaling MLPs: A Tale of Inductive Bias shows that plain MLPs follow power laws on ImageNet-21k when given heavy augmentation, but still trail CNNs and ViTs at fixed compute. We ask how much of that remaining gap is a tuning gap. Generating QM1B with PySCFIPU shows SchNet's error falling from 285 meV to 32 meV as data grows to one billion examples. It evaluates only SchNet, so whether more structured equivariant networks keep an edge at scale remains open. We fill that gap under controlled tuning.
The second group is the tuning-gap work, which supplies the method. ONCE re-tuning and the GNN pitfalls study show how to equalize tuning, but each does so at a single data size. Theoretical hyperparameter scaling (Deriving Hyperparameter Scaling Laws) predicts how optimal settings drift with training budget, for instance $\eta \propto T^{1/6}$ under joint tuning. It holds the model fixed and says nothing about whether drift differs between architectures; we measure exactly that.
Several studies sit closer to the regime-crossover framing without controlling tuning. TabLLM shows a large language model winning at 0–64 examples while trees catch up at 256 or more. Specialized Foundation Models Struggle to Beat Supervised Baselines and Period-Aware Inductive Bias Versus Scale show tuned compact models matching large pretrained ones. We do not propose a new architecture. We propose a measurement protocol and apply it to two well-defined comparisons.
Research Program
We study two testbeds where a data-size axis spanning several orders of magnitude already exists in the cited material.
In chemistry, we train SchNet against a more structured equivariant network such as PaiNN on nested QM1B subsets. The subsets run from 1M to 256M examples in factors of four, and the target is the HOMO–LUMO gap. Every model is also fine-tuned to QM9 so we can check downstream transfer.
In vision, we pretrain the B-MLP of Scaling MLPs against a ResNet and a small ViT. Pretraining uses nested ImageNet-21k subsets from 0.2M to 12M images, and evaluation is linear-probe and fine-tuned accuracy on CIFAR-10, CIFAR-100 and ImageNet ReaL.
Each (architecture, data size) cell is trained under four protocols. In the first, Fixed, we use the published default recipe at every size. In the second, Transfer, we tune at the smallest size and reuse the result. In the third, Theory, we tune at the smallest size and extrapolate learning rate, momentum and batch size with the power laws of Deriving Hyperparameter Scaling Laws. In the fourth, Per-scale, we run an independent search at each size.
Every search uses the same budget. That budget is 24 trials of random search with successive halving, over learning rate, batch size, weight decay, warmup and schedule length. Search spaces are matched in width per architecture. Each final configuration is rerun with three seeds. We fit the power law per protocol with bootstrap intervals over seeds and subsets. Beyond slopes and crossovers, we record how each optimal hyperparameter drifts with $D$ per architecture. That record directly tests whether the theory's architecture-free drift law holds across models.
Baselines
The naive reference point is the Fixed protocol. It reproduces how most scaling-law papers are run, including the shared-recipe design of Scaling Laws vs Model Architectures, so any $\Delta\alpha$ is measured relative to it.
The strongest anchor in chemistry is the published SchNet scaling result in Generating QM1B with PySCFIPU, which reports 32 meV MAE on HOMO–LUMO gap at full scale and 30.2 meV on QM9 after pretraining, against 54.13 meV from scratch. Our Fixed SchNet curve must reproduce the trend of that paper's subset results before any protocol comparison counts. Per-scale tuning "beats" it if SchNet reaches a given error with measurably less data, or if the fitted exponent moves outside its bootstrap interval.
In vision, the anchor is B-MLP at 95.5% on CIFAR-10, 82.6% on CIFAR-100 and about 58% on ImageNet ReaL, all from Scaling MLPs. The comparison is fair because every architecture gets the same trial count, seeds, augmentation pipeline and data subsets.
The ablation that isolates the key ingredient is Transfer versus Per-scale. It differs only in whether tuning is repeated at each scale, while the total search budget is matched by giving Transfer its trials at the smallest scale. The Theory protocol is the strongest cheap competitor.
Why not just…?
Why not just tune once at small scale and transfer the settings, as practitioners do? This is the Transfer arm. The ONCE study already reports that tuned settings fail to carry across benchmarks (Refining the ONCE Benchmark), which suggests they may also fail across data sizes. That is only suggestive, and if Transfer matches Per-scale within error bars, our answer is that it suffices. We would report that as a positive finding.
Why not just use the theoretical hyperparameter scaling laws instead of searching at every scale? This is the real threat. If the Theory protocol recovers per-scale exponents for all architectures, the expensive protocol is unnecessary and the field gets a cheap fix. The theory, however, holds model size fixed and contains no architectural term (Deriving Hyperparameter Scaling Laws). Our drift measurements test directly whether a single law fits SchNet and PaiNN, or B-MLP and ViT, alike.
Why not just compare fully tuned models at the largest data size, as the baseline-tuning papers do? A single point cannot locate a crossover or a slope. The QM1B result shows error changing ninefold across data sizes (Generating QM1B), and TabLLM's ranking inverts between 64 and 256 examples (TabLLM). Which model wins depends on where you look.
Why not just add more data and ignore tuning, since scale dominates? Data scale and tuning are not interchangeable. At a fixed data size, tuning alone moved ONCE by 11 points. Both effects are real, and the open question is how they interact, which is what the factorial design measures.
Proposed Experiments and Decisive Checks
The primary check compares the per-architecture $\Delta\alpha$ with its 95% bootstrap interval, and $D^*$ under Fixed against $D^*$ under Per-scale. A shift in $D^*$ of more than a factor of four, one subset step, counts as practically significant. The secondary check asks whether the fragility ordering predicted by prior work holds, meaning PaiNN and ViT showing larger $\Delta\alpha$ than SchNet and B-MLP.
The third check is downstream. Per-scale tuning must change QM9 fine-tuning MAE and CIFAR transfer accuracy in the same direction as it changes upstream error. Otherwise we have reproduced the upstream–downstream decoupling noted by Scaling Laws vs Model Architectures.
Finally, we report the cost. We give compute-hours per protocol, so that readers can see how much search the Per-scale protocol demands and whether the Theory arm is an acceptable substitute.
References
- Refining the ONCE Benchmark With Hyperparameter Tuning
- Pitfalls of Graph Neural Network Evaluation
- Deriving Hyperparameter Scaling Laws via Modern Optimization Theory
- Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
- Scaling MLPs: A Tale of Inductive Bias
- Generating QM1B with PySCFIPU
- TabLLM: Few-shot Classification of Tabular Data with Large Language Models
- Specialized Foundation Models Struggle to Beat Supervised Baselines
- Period-Aware Inductive Bias Versus Scale on Influenza-like Illness Forecasting