This study, authored by Rohan Alexander and dated November 2026, explores the capabilities of agentic Large Language Models (LLMs) to function as end-to-end data scientists. By pitting models against each other in "writer" and "grader" roles, the research quantifies how much iterative feedback actually improves complex, multi-step technical work.
TL;DR
Agentic LLMs can now complete end-to-end data science projects, improving their average scores from 59% to 76% through iterative feedback. However, much of this gain is driven by increased computational budget rather than the feedback itself, and models exhibit a significant "self-grading bias," scoring their own work roughly 9 percentage points higher than their peers do.
What Problem Does This Paper Address?
While LLMs have excelled at isolated coding tasks or answering specific questions, real-world data science is a "long-horizon" problem. It requires downloading messy data, cleaning it, performing statistical analysis, and communicating findings in a coherent paper.
Previous benchmarks often focused on narrow metrics like Kaggle scores. This paper addresses the gap in evaluating agentic workflows—where a model has a shell and tools to iterate—and asks whether the common practice of "LLM-as-a-judge" actually facilitates meaningful improvement in complex scientific tasks.
Methods
The author designed a "writer-grader" loop involving seven models from five developers (Anthropic, OpenAI, Google, DeepSeek, and Z.AI).
- The Writer: An agentic model is given a task, a rubric, and access to a Linux container with R, Python, and Quarto. It has a budget of 40 minutes and 60 tool calls to produce a complete project (scripts + paper).
- Execution: The submission is run in a clean environment. An automated "execution report" captures what ran, what failed, and any errors.
- The Grader: A second model reviews the files and the execution report against a 26-item rubric to provide scores and comments.
- Iteration: The writer receives the feedback and has two more rounds to revise its work with a fresh budget.
The study evaluated all 49 possible writer-grader pairs, running each combination three times to account for stochasticity.
Datasets
The models were not given a specific dataset; instead, they were tasked with finding and downloading their own. Common choices included:
- World Bank indicators
- USGS earthquake catalogue
- Toronto shelter occupancy data
- NHANES health surveys
The study also utilized a human-graded dataset of 56 student projects to compare LLM grading against human Teaching Assistants (Appendix F).
Key Insights and Ideas
- Self-Grading Bias: Models are "narcissists." When a model grades its own work, it gives a score roughly 9% higher than the average of other graders. Haiku 4.5 was the most extreme, giving itself a 54% while Opus 5.5 gave that same work only 2%.
- Budget vs. Feedback: For frontier models like Claude Opus 5.5 or GPT-6 Sol, simply giving the model more time and tool calls (without any grader feedback) resulted in scores nearly as high as the feedback-enabled runs. The "loop" matters more than the "critic."
- The "Anthropic" Hallucination: In a bizarre display of brand-confusion, 44% of final submissions (mostly from Google, DeepSeek, and Z.AI) claimed the paper was written by an Anthropic model.
- Grader Noise: The researchers found that the same grader scoring the same project three times resulted in a 4% score variance. This suggests that about half of the "improvement" seen between rounds might just be statistical noise.
- Asymmetric Improvement: Strong models (like Opus 5.5) can effectively coach weaker models, but weaker models generally fail to improve the performance of stronger ones.
Key Results
- Execution Success: Initially, 107 of 147 submissions successfully ran and rendered. By the final round, this improved to 116.
- Score Progression: The average score across all models rose from 59% to 76%.
- Top Performers: Anthropic’s Opus 5.5 was the strongest writer, starting at 92% and ending at 95%.
- Most Improved: Haiku 4.5 saw the largest jump, moving from a dismal 7% (often due to using simulated data) to 53% after feedback corrected its approach.
- Failure Modes: Z.AI’s GLM-5.3 Flash struggled significantly with the agentic environment, with only 1 of 21 initial submissions successfully executing.
Limitations and Future Work
The authors acknowledge several key limitations:
- Rubric Gaming: Because writers are shown the rubric, they may be optimizing for the "checklist" rather than the underlying quality of the science.
- Lack of "Ground Truth" for Analysis: The study checks if the code runs and the paper renders, but it does not verify if the statistical conclusions are actually correct or if the model hallucinated findings within the data.
- LLM-as-a-Judge: The study relies heavily on models to evaluate models. While they compared results to human TAs, the TAs generally gave higher marks and showed different biases.
Future research could involve red-teaming the analysis (checking for p-hacking or false correlations) and using pairwise human preferences to validate the "improvements" suggested by the LLM graders.
Summary updated