Recovering Wasted Compute in Autoresearch Agents

Au Kwok Chun, Abhigyan Acherjee, Amrutha Rao, Zaiqian Chen, Kazem Meidani, C. Bayan Bruss, Micah Goldblum

TL;DR

Autonomous machine learning agents (autoresearch agents) frequently waste their compute budgets by repeatedly fixing the same bugs across parallel search branches and skipping hyperparameter optimization entirely. By introducing simple agentic design interventions—a global debug consultant, budget-aware hyperparameter tuning, and Thompson Sampling-based backtracking—this paper demonstrates massive performance boosts on data science benchmarks without changing the underlying LLM.

What Problem Does This Paper Address?

LLM-powered "autoresearch" agents are increasingly being deployed to automate end-to-end data science tasks, such as writing code, training models, and evaluating machine learning pipelines. However, these agents execute tasks unreliably and waste massive amounts of compute.

When navigating a tree of potential code solutions, parallel branches suffer from context isolation. If one branch encounters a specific API bug and fixes it, other branches remain completely blind to it, wasting compute rediscovering and fixing the exact same error. Additionally, agents often stop searching prematurely once they find a single valid solution, completely skipping the hyperparameter optimization (HPO) phase, or they get trapped in dead-end debugging loops.

Methods

The authors propose three main structural interventions to recover wasted compute:

  1. Context-Aware Debug Consultant: Instead of keeping failure knowledge local to a single branch, a shared global registry compresses tracebacks into concise error signatures. It distills these into "banned patterns" and "proven fixes" and injects them into all parallel generation steps so the agent learns its runtime environment once.
Figure 1: The debug consultant shares context across the search tree: one node’s failure becomes every node’s lesson. Crashed nodes are compressed into a shared bug index of failed and successful repairs, distilled into banned patterns and proven fixes, and injected into every subsequent generation step, so the agent learns its runtime environment once instead of rediscovering each bug per branch.
  1. Budget-Aware HPO Enforcement: The authors introduce a curriculum-based control loop where an LLM judge grades the quality of hyperparameter tuning on a 0–3 scale. Early in the run, weak tuning is penalized to encourage broad exploration; later in the run, extensive tuning is heavily rewarded to force fine-grained exploitation of the best configurations.
Figure 2: Budget dependent control-loop enforcement of hyperparameter tuning in AIDE. Each node is scored for hyperparameter-tuning quality on a 0–3 scale by an LLM judge. Buggy nodes are assigned the worst possible metric and their score is ignored; for a valid node, the score both conditions the agent’s next prompt and adjusts the metric used for node selection. The reward depends on how much budget remains: weak tuning (scores of 0 or 1) is penalized throughout, but strong tuning (2 or 3) is rewarded only in later stages of the search, so the agent explores broadly early and tunes intensively late.
  1. Thompson Sampling-Enhanced Backtracking: Instead of randomly selecting code variants to expand or backtrack to, each sibling node maintains a Beta distribution tracking its historical quality. When an identical error is detected down a path, the agent backtracks to the original failure point and uses Thompson Sampling to probabilistically select the next most promising branch to explore.
Figure 3: Thompson Sampling with backtracking reallocates budget away from repeatedly failing branches. Each sibling node carries a Beta distribution over its quality, stored in global memory and updated from observed rewards after every execution. When the agent detects an identical error recurring down a path (S1 →S2 →S3), it backtracks to the branch point where the error first appeared (S1) and re-samples among the siblings, expanding the node with the highest draw rather than choosing at random. Budget that would otherwise be spent re-deriving the same failure is redirected toward more promising candidates.

Datasets

The methods were evaluated on nine tabular prediction tasks spanning classification and regression, drawn from MLE-bench and Kaggle competitions:

  • Cirrhosis Outcome Prediction
  • GNSS Classification
  • Spaceship Titanic
  • Wine Quality Ordinal
  • Playground Series (S5E3, S5E6, S5E7, S5E8, S5E12)

All tasks were selected because their release postdates the underlying LLM's (GPT-5-mini) knowledge cutoff, minimizing data leakage.

Key Insights and Ideas

  • Negative constraints are highly valuable: In reward-sparse environments like debugging, learning what not to do globally is just as important as finding a successful path.
  • Agentic scaffolds don't transfer for free: An intervention that significantly helps one agent framework can actually hurt another. For instance, forcing HPO helped the AIDE framework but degraded ML-Master. This occurred because ML-Master's specific memory architecture repeatedly retried a crashing HPO implementation without understanding why it crashed, highlighting that scaffold interventions depend heavily on internal agent architecture.
  • Agents mimic human behavior blindly: In a diagnostic test, the authors injected deliberately misleading Exploratory Data Analysis (EDA) into the context window. Current agents completely ignored the signal and failed to let it impact downstream decisions, highlighting a lack of true data-driven planning loops.

Key Results

  • Debug Consultant Performance: The debug consultant nearly doubled the gold-medal count for the AIDE framework (from 22 to 38) and eliminated all failed runs, boosting the valid submission rate from 81% to 100%. ML-Master saw a similar jump from 18 to 29 gold medals.
The debug consultant improves gold medal rates on both agents
  • Hyperparameter Guidance: Explicit HPO guidance yielded massive performance gains for AIDE across 7 out of 9 competitions, with individual score improvements up to +0.388 on specific tasks.
Hyperparameter guidance yields large performance gains for AIDE
  • Thompson Sampling Stability: Replacing random sibling selection with Thompson Sampling more than halved the number of null runs (runs that fail to produce any valid submission), drastically improving search stability at no cost to peak performance.

Limitations and Future Work

  • Scaffold Asymmetry: Interventions must be carefully tailored to an agent's specific memory and reasoning architecture, as shown by the HPO degradation in ML-Master.
  • Lack of True Exploration: Tree-search algorithms assume expanding a node produces a novel candidate, but LLMs often write nearly the same program over and over. Future work must design agents that propose genuinely diverse solutions.
  • EDA Blindness: Current agents do not meaningfully act on or integrate exploratory data analysis into downstream modeling decisions, representing a major gap in autonomous scientific reasoning.

Summary updated

Abstract

A slew of recent works develop agents for solving research problems end-to-end, a paradigm increasingly referred to as autoresearch. Such agents have inspired large industry investment, motivated by their potential to automate time-consuming human labor and customize machine learning solutions for specialized applications. In this paper, we study the modeling pipeline at the core of these autoresearch systems and identify common failure modes when they are applied to tabular datasets: (1) they waste compute resolving the same bugs over and over again; (2) they often fail to tune hyperparameters even when they have a large remaining compute budget; (3) the tree-search algorithms that power them do not explore; and (4) they perform data analysis, mimicking the humans whose data they are trained on, but do not use that analysis to make downstream decisions. We explore targeted interventions and find that a global debug consultant that shares discovered runtime constraints across all branches of the search tree, prompt- and control-level enhancements, and refined tree-search algorithms successfully recover wasted compute. Our results show that large gains in autoresearch agent performance are achievable through agentic design alone, holding the underlying language model fixed.
BibTeX
@inproceedings{chun2026recovering,
  author = {Au Kwok Chun and Abhigyan Acherjee and Amrutha Rao and Zaiqian Chen and Kazem Meidani and C. Bayan Bruss and Micah Goldblum},
  title = {{Recovering Wasted Compute in Autoresearch Agents}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=F5ajYXqGwb},
}
Figure