HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution

Hanhua Hong, Yizhi Li, Jiaoyan Chen, Sophia Ananiadou, Xiaoli Li, Jung-jae Kim, Chenghua Lin

TL;DR

The paper introduces HiRAS (Hierarchical Research Agent System), a multi-agent framework designed to automate the end-to-end reproduction of scientific papers into executable code repositories. By replacing rigid sequential pipelines with proactive supervisory manager agents, HiRAS achieves state-of-the-art reproduction fidelity and robustness. Additionally, the authors expose critical flaws in existing reference-free LLM evaluation metrics and propose Paper2Code-Extra (P2C-Ex), a significantly more reliable evaluation protocol.

What Problem Does This Paper Address?

Reproducing experimental results from published papers is a cornerstone of scientific progress, yet it is an incredibly labor-intensive task. On average, only about 20% of papers release complete, usable code repositories. While large language models (LLMs) show promise in automating code generation, existing multi-agent frameworks rely on fixed, sequential pipelines. In these setups, an error introduced by an agent early in the process (e.g., during planning) propagates unchecked, causing the entire workflow to stall or produce incomplete repositories.

Furthermore, evaluating these automated systems is notoriously difficult. Current benchmarks often rely on reference-free LLM-as-a-judge metrics due to the complexity of the task. However, the authors discover that these evaluators are highly prone to hallucination—frequently awarding high scores to completely empty or configuration-only repositories because they infer code existence from documentation alone.

Methods

HiRAS addresses coordination failures by introducing a tree-structured hierarchical management architecture. Instead of a flat pipeline, the workflow is split into fine-grained phases handled by specialized agents:

  • Planning Agents: Decomposed into Overall Planning, Architecture Design, Dependency Modelling, and Configuration Generation.
  • Analysis & Coding Agents: Specialized units that generate implementation details and write the actual codebase.
  • Execution Agent: An agent equipped with a system console (bash tool) to run the code, validate executability, and collect logs.

Crucially, HiRAS introduces Supervisory Manager Agents (a Global Manager and a Planning Manager) that actively oversee the process. Unlike passive message-passing managers in prior work, these managers proactively inspect intermediate artifacts in a shared workspace, diagnose failures, and dynamically re-invoke subordinate agents with corrective feedback if errors or omissions are detected.

Overview of the HiRAS framework

To fix the evaluation problem, the authors introduce Paper2Code-Extra (P2C-Ex). This refined protocol injects repository-level context—such as total file count and a hierarchical repository structure illustration—directly into the evaluator's prompt, alongside strict instructions to prevent the LLM judge from hallucinating non-existent code.

Datasets

The framework was evaluated on two primary experiment reproduction benchmarks:

  1. PaperBench: Consists of 20 machine learning papers from ICML 2024, complete with detailed, paper-specific manual evaluation rubrics.
  2. Paper2Code: Contains 90 papers collected from ICML 2024, NeurIPS 2024, and ICLR 2024.

Key Insights and Ideas

  • Proactive Supervision Beats Fixed Pipelines: Decomposing a task is not enough; agents need dynamic error correction. Allowing a manager agent to inspect outputs and re-invoke coding or planning agents prevents early-stage errors from ruining the final execution.
  • LLM Judges are Easily Fooled by Absence: Standard reference-free prompts cause LLMs to hallucinate code that isn't there. For instance, an empty repository scored an astonishing 3.89 out of 5 under the original Paper2Code metric because the judge inferred implementation from paper text alone.
  • Structural Context Anchors LLM Evaluators: Providing explicit structural metadata (like file counts and directory trees) to an LLM judge dramatically grounds its evaluation, aligning reference-free metrics closely with expensive reference-based gold standards.

Key Results

  • State-of-the-Art Performance: On the PaperBench-CodeDev subset, HiRAS achieved a score of 64.1% using Claude-Sonnet, significantly outperforming prior frameworks like PaperCoder (51.1%) and AutoReproduce (48.5%).
  • Strong Open-Source Capabilities: When using open-source backbones, HiRAS achieved 57.4% with DeepSeek-v3.1-Terminus, surpassing the previous SOTA that relied on proprietary models.
Main results on the PaperBench-CodeDev subset
  • Dramatically Improved Evaluation Alignment: The proposed P2C-Ex protocol increased the Pearson correlation coefficient ($r$) with the gold-standard reference-based metric from a weak 0.423 to a highly reliable 0.862, successfully mitigating evaluator hallucination.

Limitations and Future Work

  • Execution Bottlenecks: While code generation is highly robust, the actual execution stage remains a major bottleneck. The system still struggles with complex inter-file dependencies, environment setup failures, and downloading external datasets/models from platforms like GitHub or Hugging Face.
  • Increased Overhead: The hierarchical structure and iterative self-refinement loops incur significantly higher time and token costs compared to simple sequential pipelines.
  • Evaluation Scope: Due to budget constraints, not all experimental configurations could be evaluated using premium models like o3-mini.

Summary updated

Abstract

Recent advances in large language models have highlighted their potential to automate computational research, particularly reproducing experimental results. However, existing approaches still use fixed sequential agent pipelines with weak global coordination, which limits their robustness and overall performance. In this work, we propose Hierarchical Research Agent System (HiRAS), a hierarchical multi-agent framework for end-to-end experiment reproduction that employs supervisory manager agents to coordinate specialised agents across fine-grained stages. We also identify limitations in the reference-free evaluation of the Paper2Code benchmark and introduce Paper2Code-Extra (P2C-Ex), a refined protocol that incorporates repository-level information and better aligns with the original reference-based metric. We conduct extensive evaluation, validating the effectiveness and robustness of our proposed methods, and observing improvements, including >10\% relative performance gain beyond the previous state-of-the-art using open-source backbone models and significantly reduced hallucination in evaluation. Our work is available on GitHub: https://github.com/KOU-199024/HiRAS.
BibTeX
@inproceedings{hong2026hiras,
  author = {Hanhua Hong and Yizhi LI and Jiaoyan Chen and Sophia Ananiadou and Xiaoli Li and Jung-jae Kim and Chenghua Lin},
  title = {{HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.377},
}

Loading related content...

Figure