TL;DR
The paper introduces HiRAS (Hierarchical Research Agent System), a multi-agent framework designed to automate the end-to-end reproduction of scientific papers into executable code repositories. By replacing rigid sequential pipelines with proactive supervisory manager agents, HiRAS achieves state-of-the-art reproduction fidelity and robustness. Additionally, the authors expose critical flaws in existing reference-free LLM evaluation metrics and propose Paper2Code-Extra (P2C-Ex), a significantly more reliable evaluation protocol.
What Problem Does This Paper Address?
Reproducing experimental results from published papers is a cornerstone of scientific progress, yet it is an incredibly labor-intensive task. On average, only about 20% of papers release complete, usable code repositories. While large language models (LLMs) show promise in automating code generation, existing multi-agent frameworks rely on fixed, sequential pipelines. In these setups, an error introduced by an agent early in the process (e.g., during planning) propagates unchecked, causing the entire workflow to stall or produce incomplete repositories.
Furthermore, evaluating these automated systems is notoriously difficult. Current benchmarks often rely on reference-free LLM-as-a-judge metrics due to the complexity of the task. However, the authors discover that these evaluators are highly prone to hallucination—frequently awarding high scores to completely empty or configuration-only repositories because they infer code existence from documentation alone.
Methods
HiRAS addresses coordination failures by introducing a tree-structured hierarchical management architecture. Instead of a flat pipeline, the workflow is split into fine-grained phases handled by specialized agents:
- Planning Agents: Decomposed into Overall Planning, Architecture Design, Dependency Modelling, and Configuration Generation.
- Analysis & Coding Agents: Specialized units that generate implementation details and write the actual codebase.
- Execution Agent: An agent equipped with a system console (
bashtool) to run the code, validate executability, and collect logs.
Crucially, HiRAS introduces Supervisory Manager Agents (a Global Manager and a Planning Manager) that actively oversee the process. Unlike passive message-passing managers in prior work, these managers proactively inspect intermediate artifacts in a shared workspace, diagnose failures, and dynamically re-invoke subordinate agents with corrective feedback if errors or omissions are detected.
To fix the evaluation problem, the authors introduce Paper2Code-Extra (P2C-Ex). This refined protocol injects repository-level context—such as total file count and a hierarchical repository structure illustration—directly into the evaluator's prompt, alongside strict instructions to prevent the LLM judge from hallucinating non-existent code.
Datasets
The framework was evaluated on two primary experiment reproduction benchmarks:
- PaperBench: Consists of 20 machine learning papers from ICML 2024, complete with detailed, paper-specific manual evaluation rubrics.
- Paper2Code: Contains 90 papers collected from ICML 2024, NeurIPS 2024, and ICLR 2024.
Key Insights and Ideas
- Proactive Supervision Beats Fixed Pipelines: Decomposing a task is not enough; agents need dynamic error correction. Allowing a manager agent to inspect outputs and re-invoke coding or planning agents prevents early-stage errors from ruining the final execution.
- LLM Judges are Easily Fooled by Absence: Standard reference-free prompts cause LLMs to hallucinate code that isn't there. For instance, an empty repository scored an astonishing 3.89 out of 5 under the original Paper2Code metric because the judge inferred implementation from paper text alone.
- Structural Context Anchors LLM Evaluators: Providing explicit structural metadata (like file counts and directory trees) to an LLM judge dramatically grounds its evaluation, aligning reference-free metrics closely with expensive reference-based gold standards.
Key Results
- State-of-the-Art Performance: On the PaperBench-CodeDev subset, HiRAS achieved a score of 64.1% using Claude-Sonnet, significantly outperforming prior frameworks like PaperCoder (51.1%) and AutoReproduce (48.5%).
- Strong Open-Source Capabilities: When using open-source backbones, HiRAS achieved 57.4% with DeepSeek-v3.1-Terminus, surpassing the previous SOTA that relied on proprietary models.
- Dramatically Improved Evaluation Alignment: The proposed P2C-Ex protocol increased the Pearson correlation coefficient ($r$) with the gold-standard reference-based metric from a weak 0.423 to a highly reliable 0.862, successfully mitigating evaluator hallucination.
Limitations and Future Work
- Execution Bottlenecks: While code generation is highly robust, the actual execution stage remains a major bottleneck. The system still struggles with complex inter-file dependencies, environment setup failures, and downloading external datasets/models from platforms like GitHub or Hugging Face.
- Increased Overhead: The hierarchical structure and iterative self-refinement loops incur significantly higher time and token costs compared to simple sequential pipelines.
- Evaluation Scope: Due to budget constraints, not all experimental configurations could be evaluated using premium models like
o3-mini.
Summary updated