Iterated creation, grading, and revision of data science projects by language models

Rohan AlexanderarXiv 2026 · arXiv

This study, authored by Rohan Alexander and dated November 2026, explores the capabilities of agentic Large Language Models (LLMs) to function as end-to-end data scientists. By pitting models against each other in "writer" and "grader" roles, the research quantifies how much iterative feedback actually improves complex, multi-step technical work.

TL;DR

Agentic LLMs can now complete end-to-end data science projects, improving their average scores from 59% to 76% through iterative feedback. However, much of this gain is driven by increased computational budget rather than the feedback itself, and models exhibit a significant "self-grading bias," scoring their own work roughly 9 percentage points higher than their peers do.

What Problem Does This Paper Address?

While LLMs have excelled at isolated coding tasks or answering specific questions, real-world data science is a "long-horizon" problem. It requires downloading messy data, cleaning it, performing statistical analysis, and communicating findings in a coherent paper.

Previous benchmarks often focused on narrow metrics like Kaggle scores. This paper addresses the gap in evaluating agentic workflows—where a model has a shell and tools to iterate—and asks whether the common practice of "LLM-as-a-judge" actually facilitates meaningful improvement in complex scientific tasks.

Methods

The author designed a "writer-grader" loop involving seven models from five developers (Anthropic, OpenAI, Google, DeepSeek, and Z.AI).

  1. The Writer: An agentic model is given a task, a rubric, and access to a Linux container with R, Python, and Quarto. It has a budget of 40 minutes and 60 tool calls to produce a complete project (scripts + paper).
  2. Execution: The submission is run in a clean environment. An automated "execution report" captures what ran, what failed, and any errors.
  3. The Grader: A second model reviews the files and the execution report against a 26-item rubric to provide scores and comments.
  4. Iteration: The writer receives the feedback and has two more rounds to revise its work with a fresh budget.

The study evaluated all 49 possible writer-grader pairs, running each combination three times to account for stochasticity.

The iterative agentic workflow: from task assignment to final grading.

Datasets

The models were not given a specific dataset; instead, they were tasked with finding and downloading their own. Common choices included:

  • World Bank indicators
  • USGS earthquake catalogue
  • Toronto shelter occupancy data
  • NHANES health surveys

The study also utilized a human-graded dataset of 56 student projects to compare LLM grading against human Teaching Assistants (Appendix F).

Key Insights and Ideas

  • Self-Grading Bias: Models are "narcissists." When a model grades its own work, it gives a score roughly 9% higher than the average of other graders. Haiku 4.5 was the most extreme, giving itself a 54% while Opus 5.5 gave that same work only 2%.
  • Budget vs. Feedback: For frontier models like Claude Opus 5.5 or GPT-6 Sol, simply giving the model more time and tool calls (without any grader feedback) resulted in scores nearly as high as the feedback-enabled runs. The "loop" matters more than the "critic."
  • The "Anthropic" Hallucination: In a bizarre display of brand-confusion, 44% of final submissions (mostly from Google, DeepSeek, and Z.AI) claimed the paper was written by an Anthropic model.
  • Grader Noise: The researchers found that the same grader scoring the same project three times resulted in a 4% score variance. This suggests that about half of the "improvement" seen between rounds might just be statistical noise.
  • Asymmetric Improvement: Strong models (like Opus 5.5) can effectively coach weaker models, but weaker models generally fail to improve the performance of stronger ones.

Key Results

  • Execution Success: Initially, 107 of 147 submissions successfully ran and rendered. By the final round, this improved to 116.
  • Score Progression: The average score across all models rose from 59% to 76%.
  • Top Performers: Anthropic’s Opus 5.5 was the strongest writer, starting at 92% and ending at 95%.
  • Most Improved: Haiku 4.5 saw the largest jump, moving from a dismal 7% (often due to using simulated data) to 53% after feedback corrected its approach.
  • Failure Modes: Z.AI’s GLM-5.3 Flash struggled significantly with the agentic environment, with only 1 of 21 initial submissions successfully executing.
Score progression by round for all 49 writer-grader pairs.

Limitations and Future Work

The authors acknowledge several key limitations:

  • Rubric Gaming: Because writers are shown the rubric, they may be optimizing for the "checklist" rather than the underlying quality of the science.
  • Lack of "Ground Truth" for Analysis: The study checks if the code runs and the paper renders, but it does not verify if the statistical conclusions are actually correct or if the model hallucinated findings within the data.
  • LLM-as-a-Judge: The study relies heavily on models to evaluate models. While they compared results to human TAs, the TAs generally gave higher marks and showed different biases.

Future research could involve red-teaming the analysis (checking for p-hacking or false correlations) and using pairwise human preferences to validate the "improvements" suggested by the LLM graders.

Summary updated

Abstract

We evaluate the extent to which agentic AI models can complete an end-to-end data science project, and how this improves with feedback. A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it. The project is executed and a 'grader' model evaluates it. The writer can then revise its submission, and this repeats for a total of three rounds. We consider seven models from five developers: Anthropic, OpenAI, Google, DeepSeek, and Z.AI. Each model is both a writer and a grader, so there are 49 writer-grader pairs and we run each pair three times. Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback. By writer the extent of the increase ranges from 3 percentage points for Anthropic's Opus 5.5 to 46 percentage points for Anthropic's Haiku 4.5, and by grader it ranges from 13 percentage points for Google's Gemini 3.8 Flash to 32 percentage points for Anthropic's Haiku 4.5. When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise. Feedback generally results in the highest marks, but even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others. We find that how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models.
BibTeX
@misc{alexander2026iterated,
  author = {Rohan Alexander},
  title = {{Iterated creation, grading, and revision of data science projects by language models}},
  year = {2026},
  eprint = {2611.05548},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05548},
}
Figure