TL;DR
LLM agents for long-horizon engineering tasks eventually "tire out," suffering from declining reasoning intensity and stalling progress. This paper introduces Humanize MLE Agents (HMA), a framework that alternates between two different agents on a shared workspace, resetting their conversation context every few steps to maintain "fresh eyes." This simple workflow-level coordination achieves state-of-the-art results on the MLE-bench leaderboard, outperforming complex single-agent systems while using significantly less time.
What Problem Does This Paper Address?
Building autonomous agents for Machine Learning Engineering (MLE)—tasks like winning a Kaggle competition—requires long-horizon reasoning. Current approaches rely on "context engineering" or custom "harnesses" (specialized prompts and tool-management systems). However, the authors identify two critical failure modes:
- Long-horizon Deceleration: As a single-agent session continues, the "reasoning intensity" (measured by output tokens per response) drops significantly, and performance gains plateau. The agent essentially gets bogged down by its own history.
- Harness Staleness: Specialized harnesses designed to help a specific model often become redundant or even harmful when that model is upgraded. A harness that helps a "weaker" model might hinder a "stronger" one that no longer needs those specific crutches.
The core question is: Can we move beyond these single-agent limits by coordinating multiple, diverse agents at the workflow level rather than just engineering better prompts for one?
Methods
The authors propose Humanize MLE Agents (HMA), which treats agents as "black boxes" and coordinates them through a structured alternation strategy:
- Multi-Agent Alternation: The system alternates between two different agents (e.g., GPT-4o and Claude 3.5 Sonnet).
- Fixed-k Interruption: Each agent session is capped at a maximum of $k$ experiments (the authors found $k=5$ to be effective).
- Context Reset with Shared Workspace: When agents switch, the conversation context is completely reset to $\emptyset$. However, the agents share a persistent workspace (code, data, logs, and trained models). This forces the incoming agent to re-evaluate the current state of the project from scratch, often discovering refinements the previous agent missed.
- Theoretical Framework: The authors model this process as a context-reset semi-Markov decision process (CR-SMDP). This allows them to mathematically prove that alternating complementary agents can exceed the performance ceiling of any single agent.
- Final Review: At the end of the time budget, a final agent reviews the archive of all generated candidates to select the best one for submission.
Datasets
The primary evaluation was conducted on MLE-bench, a benchmark consisting of 75 Kaggle competitions. Tasks range from "Lite" (easier) to "High" complexity.
To demonstrate the framework's generality beyond MLE, the authors also tested it on AOPT (Anthropic's Original Performance Take-Home), a challenging kernel optimization task.
Key Insights and Ideas
- Reasoning as a Resource: The paper uses mean output tokens per response as a proxy for reasoning effort. They observed that in single-agent runs, this effort drops by 30-45% over a 6-hour window. HMA, by contrast, retains 86.4% of its reasoning intensity by constantly "refreshing" the session.
- The "Fresh Eyes" Effect: Resetting the context prevents an agent from getting stuck in a "local optimum" of reasoning. A new agent, seeing only the code and results but not the previous agent's internal monologue, is more likely to spot errors or new directions.
- Workflow over Harness: Instead of building increasingly complex, brittle harnesses, the authors show that using a model's "native" harness and focusing on the workflow of how agents hand off tasks is more robust and effective.
Key Results
- State-of-the-Art Performance: On MLE-bench, HMA achieved an any-medal rate of 78.2%, surpassing the strongest open-source baseline by 8.0 percentage points.
- Efficiency: HMA reached these results using only a quarter of the time budget required by some competing specialized workflows.
- Cross-Pair Generality: The benefit wasn't limited to a specific pair of models. Across six different configurations involving five different model families, HMA improved the medal rate by an average of 6.4 points over the best constituent single agent.
- Beyond MLE: In the AOPT kernel optimization task, HMA reached 878 cycles, placing it within 1.6% of the top of the community leaderboard.
Limitations and Future Work
- The Handoff Tax: There is an inherent cost to resetting context; the new agent must spend some time "re-learning" the workspace. While the "fresh eyes" benefit currently outweighs this cost, optimizing the handoff (perhaps by providing a brief summary) is a potential area for improvement.
- Fixed vs. Adaptive Caps: The current system uses a fixed $k$ (number of experiments). Future work could explore adaptive termination—switching agents only when progress is detected to be slowing down.
- Scaling to $N$ Agents: This study focused on pairs. Exploring how to coordinate larger "teams" of heterogeneous agents with different specialized roles is a natural next step.
- Dependency on Artifacts: HMA relies on a shared workspace with persistent artifacts (code, files). It remains to be seen how well this generalizes to long-horizon tasks that are purely conversational or lack a persistent file system.
Summary updated