TL;DR
Deep learning progress in the sciences is often overstated because researchers fail to compare their complex models against strong, simple baselines. This paper demonstrates that Large Language Models (LLMs) can automatically construct these missing baselines, which recover a median of 95.5% of the performance gains reported in top-tier journals like Nature and Science—matching or beating the published deep learning model in one-third of the cases.
What Problem Does This Paper Address?
In the rush to apply deep learning to the physical and life sciences, a critical step is often skipped: the "sanity check." To prove a novel, complex architecture is actually better, it must be compared against a "simple baseline"—like a well-tuned random forest or a linear model.
However, building these baselines takes time and domain expertise that human reviewers often lack. Consequently, the literature is full of "illusory progress," where a complex model appears revolutionary only because the comparison point was a "trivial" predictor (like always guessing the average) rather than a competent, simpler alternative. The authors investigate whether LLMs can serve as an automated, scalable auditor to fill this gap.
Methods
The authors developed an LLM-driven workflow using Claude Opus 5 to act as a machine learning engineer.
- The Harness: The agent operates in a containerized environment with no internet access, no GPU, and no pre-trained weights. It is given the task's training data, evaluation metrics, and varying levels of study context (from the full paper text to just the column names).
- The Task: The agent must write and execute a Python script within a 30-minute window to train a baseline model (typically using libraries like Scikit-learn, XGBoost, or PyTorch) and submit predictions.
- The Metric: They introduced Simple Baseline Strength (SBS). This metric scales performance: 0 represents a "trivial floor" (e.g., predicting the mean), and 1 represents the performance of the published deep learning (DL) method. An SBS $\ge$ 1 means the automated baseline matched or outperformed the complex DL model.
Datasets
The study assembled a primary cohort of 36 supervised classification and regression tasks extracted from peer-reviewed studies published between 2024 and 2026 in Nature, Science, Cell, and their sister journals. These tasks span eight scientific domains, including:
- Neuroscience (e.g., spike inference)
- Medicinal chemistry (e.g., bioactivity prediction)
- Structural biology (e.g., protein-ligand binding)
- Genomics (e.g., regulatory activity)
A secondary cohort of 9 tasks was used for deeper dives into how test-time compute and context availability affect performance.
Key Insights and Ideas
- The "Complexity Tax" is often unnecessary: The fact that automated, simple baselines (often gradient-boosted trees) can recover 95.5% of the performance of bespoke deep learning models suggests that many scientific problems don't yet require the massive architectural complexity currently being published.
- LLMs are competent ML Engineers: The agent didn't just guess; it successfully wrote valid code, handled data loading, and performed internal validation. In the primary cohort, 157 out of 180 attempts produced scorable results.
- Context vs. Data: As shown in the ablation studies, the LLM's performance is highly dependent on having the training data. However, it is surprisingly robust to the loss of the full paper text; having just the abstract and the data was often enough to build a strong baseline.
- Scalable Peer Review: This tool could act as a "red team" for journals, providing reviewers with an immediate estimate of how much a new method actually contributes beyond standard ML techniques.
Key Results
- Overall Strength: Across 36 tasks, the automated baselines achieved a median SBS of 95.5% (mean 85.7%).
- Beating the "State-of-the-Art": In 12 out of 36 tasks, the median automated baseline matched or exceeded the performance of the published deep learning model.
- Domain Variation: Performance varied by field. In Neuroscience, the automated baselines were exceptionally strong (mean SBS 1.08), while in Genomics, they were relatively weaker (mean SBS 0.73), though still significant.
- Ablation Findings: Removing training data caused the most significant drop in performance. When the agent had no training data (L6), SBS dropped to a median of 0.17.
Limitations and Future Work
- Task Scope: The study only looked at supervised regression and classification. It did not evaluate generative modeling or representation learning, which are more complex to benchmark.
- Model Selection: The study relied solely on Claude Opus 5. Results might differ with other LLMs.
- Data Leakage: There is a risk that the LLM saw the source papers or datasets during its pre-training, though the authors attempted to mitigate this by using very recent papers and redacting specific results in some experiments.
- Trivial Predictor Sensitivity: The SBS metric is sensitive to the definition of the "trivial floor." A different choice of trivial predictor could shift the results.
- Future Directions: The authors suggest expanding this to more complex tasks and integrating automated baseline construction directly into the scientific publishing pipeline.
Summary updated