Automatic Construction of Strong Simple Baselines

Caleb N. Ellington, Jiaqi Wang, Sohan Addagudi, Elijah ColearXiv 2026 · arXiv

TL;DR

Deep learning progress in the sciences is often overstated because researchers fail to compare their complex models against strong, simple baselines. This paper demonstrates that Large Language Models (LLMs) can automatically construct these missing baselines, which recover a median of 95.5% of the performance gains reported in top-tier journals like Nature and Science—matching or beating the published deep learning model in one-third of the cases.

What Problem Does This Paper Address?

In the rush to apply deep learning to the physical and life sciences, a critical step is often skipped: the "sanity check." To prove a novel, complex architecture is actually better, it must be compared against a "simple baseline"—like a well-tuned random forest or a linear model.

However, building these baselines takes time and domain expertise that human reviewers often lack. Consequently, the literature is full of "illusory progress," where a complex model appears revolutionary only because the comparison point was a "trivial" predictor (like always guessing the average) rather than a competent, simpler alternative. The authors investigate whether LLMs can serve as an automated, scalable auditor to fill this gap.

Methods

The authors developed an LLM-driven workflow using Claude Opus 5 to act as a machine learning engineer.

  1. The Harness: The agent operates in a containerized environment with no internet access, no GPU, and no pre-trained weights. It is given the task's training data, evaluation metrics, and varying levels of study context (from the full paper text to just the column names).
  2. The Task: The agent must write and execute a Python script within a 30-minute window to train a baseline model (typically using libraries like Scikit-learn, XGBoost, or PyTorch) and submit predictions.
  3. The Metric: They introduced Simple Baseline Strength (SBS). This metric scales performance: 0 represents a "trivial floor" (e.g., predicting the mean), and 1 represents the performance of the published deep learning (DL) method. An SBS $\ge$ 1 means the automated baseline matched or outperformed the complex DL model.

Datasets

The study assembled a primary cohort of 36 supervised classification and regression tasks extracted from peer-reviewed studies published between 2024 and 2026 in Nature, Science, Cell, and their sister journals. These tasks span eight scientific domains, including:

  • Neuroscience (e.g., spike inference)
  • Medicinal chemistry (e.g., bioactivity prediction)
  • Structural biology (e.g., protein-ligand binding)
  • Genomics (e.g., regulatory activity)

A secondary cohort of 9 tasks was used for deeper dives into how test-time compute and context availability affect performance.

Key Insights and Ideas

  • The "Complexity Tax" is often unnecessary: The fact that automated, simple baselines (often gradient-boosted trees) can recover 95.5% of the performance of bespoke deep learning models suggests that many scientific problems don't yet require the massive architectural complexity currently being published.
  • LLMs are competent ML Engineers: The agent didn't just guess; it successfully wrote valid code, handled data loading, and performed internal validation. In the primary cohort, 157 out of 180 attempts produced scorable results.
  • Context vs. Data: As shown in the ablation studies, the LLM's performance is highly dependent on having the training data. However, it is surprisingly robust to the loss of the full paper text; having just the abstract and the data was often enough to build a strong baseline.
  • Scalable Peer Review: This tool could act as a "red team" for journals, providing reviewers with an immediate estimate of how much a new method actually contributes beyond standard ML techniques.

Key Results

  • Overall Strength: Across 36 tasks, the automated baselines achieved a median SBS of 95.5% (mean 85.7%).
  • Beating the "State-of-the-Art": In 12 out of 36 tasks, the median automated baseline matched or exceeded the performance of the published deep learning model.
Figure 1: Simple baseline strength across 36 tasks. The bars show how much of the DL model's gain was recovered by the LLM agent. Note that many bars reach or cross the 1.0 (DL method) line.
  • Domain Variation: Performance varied by field. In Neuroscience, the automated baselines were exceptionally strong (mean SBS 1.08), while in Genomics, they were relatively weaker (mean SBS 0.73), though still significant.
Figure 2: Simple baseline strength by scientific field for the cohort of Figure 1, over groups of three or more tasks. Dots are each task’s median SBS score. Bars show the group mean over these task medians. The upper line marks the published deep learning method’s performance (strength = 1) and the lower line is the trivial predictor’s performance (strength = 0). Task counts follow the group names.
  • Ablation Findings: Removing training data caused the most significant drop in performance. When the agent had no training data (L6), SBS dropped to a median of 0.17.
Figure 5: Simple baseline strength with Claude Opus 5 under each context ablation on the secondary cohort (Section 3.1). Dots are each task’s median SBS over its scored attempts. Bars show the median over tasks. All values lie within the axis. Appendix Table 3 lists what every level receives. Significance is determined relative to L1 using two-sided paired Wilcoxon signed-rank tests with Holm correction. ∗: p < 0.05, ∗∗: p < 0.01, ∗∗∗: p < 0.001, ns: not significant. Runs without a scored attempt are excluded (Appendix D.2).

Limitations and Future Work

  • Task Scope: The study only looked at supervised regression and classification. It did not evaluate generative modeling or representation learning, which are more complex to benchmark.
  • Model Selection: The study relied solely on Claude Opus 5. Results might differ with other LLMs.
  • Data Leakage: There is a risk that the LLM saw the source papers or datasets during its pre-training, though the authors attempted to mitigate this by using very recent papers and redacting specific results in some experiments.
  • Trivial Predictor Sensitivity: The SBS metric is sensitive to the definition of the "trivial floor." A different choice of trivial predictor could shift the results.
  • Future Directions: The authors suggest expanding this to more complex tasks and integrating automated baseline construction directly into the scientific publishing pipeline.

Summary updated

Abstract

To rigorously establish the value of a novel deep learning method, it must be compared against appropriate baselines. As deep learning has become increasingly common in the physical and life sciences, there have been numerous cases in which apparent progress has turned out to be an artifact of omitted or missing baselines. In this work, we investigate whether this problem can be mitigated by using large language models to automate the construction of baselines. We assemble 36 supervised classification and regression tasks from peer-reviewed studies in Nature, Science, and Cell-family journals that investigate novel deep learning methods for scientific applications. We then use a language model to construct simple baselines for each task. These automatically constructed baselines are strong, recovering a median of 95.5% (mean 85.7%) of the gap between a trivial predictor and the score of the deep learning model. The median simple baseline matches or exceeds the performance of the deep learning model in 12 studies. We further examine the variation across scientific domains and publication venues and we characterize the effects of test-time compute and access to different levels of study context. As human peer review has struggled to keep up with the rapidly increasing volume of research articles in recent years, our results suggest that automated baseline construction is a practical and scalable check on deep learning contributions in the physical and life sciences.
BibTeX
@misc{ellington2026automatic,
  author = {Caleb N. Ellington and Jiaqi Wang and Sohan Addagudi and Elijah Cole},
  title = {{Automatic Construction of Strong Simple Baselines}},
  year = {2026},
  eprint = {2610.40700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40700},
}
Figure