Unified Software Engineering agent as AI Software Engineer

Nan Jiang, Leonhard Applis, Yuntong Zhang, Shanchao Liang, Nan Jiang 0012, Lin Tan, Abhik RoychoudhuryarXiv 2025 · DOI · arXiv

Abstract

Recent advancements in AI for software engineering have largely focused on specialized agents for specific tasks like bug fixing or test generation. This paper introduces USEagent, a unified AI agent designed as a generalist software engineer. By employing a central "Meta-Agent" to orchestrate modular actions, USEagent achieves a 33.3% success rate across the new USEbench—a massive 1,271-task benchmark. The agent matches the performance of specialized tools while handling a significantly broader range of engineering challenges through dynamic workflow construction.

Introduction

The current landscape of AI-driven software engineering is fragmented, characterized by "specialist" agents optimized for narrow tasks such as program repair or unit test generation. However, human software engineers are generalists who navigate projects, reproduce bugs, and implement features interchangeably. Existing agents often rely on rigid, fixed workflows that struggle with tasks outside their specific design.

To bridge this gap, researchers developed USEagent. The core research question asks: Can we build a unified agent that autonomously decides which tools to use and which workflows to construct on the fly?

Methods: The Meta-Agent Architecture

USEagent moves away from hardcoded pipelines toward an autonomous orchestration model. Its architecture is built upon three pillars:

  1. The Meta-Agent: A central LLM-based reasoning engine (e.g., Claude 3.5 Sonnet) operating in a ReAct-style loop. It reasons about the current state and selects the next action dynamically.
  2. Modular Actions: Encapsulated "units of work" including CodeRetrieval, Reproduction, EditCode, and ExecuteTests.
  3. Task State (Consensus Memory): A structured shared memory tracking code locations, test results, and version diffs, ensuring continuity across different phases of a task.
The USEagent architecture featuring the Meta-Agent and its modular Action Space.

By disassembling fixed workflows, USEagent self-configures its path based on the task at hand. For instance, it might prioritize reproduction for a bug fix but lead with code retrieval for a new feature.

USEbench: A Unified Benchmark

The authors introduce USEbench, a "meta-benchmark" comprising 1,271 repository-level tasks. It aggregates several existing specialized benchmarks into a single unified interface to test generalist capabilities.

Summary of the primitive and compound tasks included in the USEbench meta-benchmark.

The benchmark includes:

  • SWE-bench (Verified): Real-world GitHub issues for program repair.
  • SWT-bench: Regression testing tasks.
  • REPOCOD & REPOTEST: Repository-level code and test generation.
  • Compound Tasks: Complex scenarios like "Partial Fix" and "Feature Development."

Key Insights and Results

USEagent demonstrates that a generalist approach does not necessarily sacrifice efficacy. On the SWE-Verified benchmark, it achieved a 45.6% success rate, nearly matching the 46.2% of AutoCodeRover, a tool specifically optimized for that task.

Dynamic Workflows and Recovery

A critical finding was that dynamic workflows performed 4% to 33% better than static sequences. The Meta-Agent's ability to "reason" its way out of sub-optimal initial decisions—such as trying a different retrieval strategy if the first fails—is a significant advantage over rigid pipelines.

Performance Metrics

  • Overall Efficacy: USEagent solved 33.3% of all USEbench tasks, significantly outperforming general-purpose agents like OpenHands CodeActAgent (26.8%).
  • Task-Specific Success: The agent achieved 40.3% success in reproduction and 31.8% in test generation. However, full-method generation (REPOCOD) remains a challenge at 6.0%.
  • Cost Efficiency: Using open-source models like DeepSeek-V3 reduced costs by approximately 85% while maintaining competitive performance.

Limitations and Future Directions

Despite its success, USEagent faces challenges with effective backtracking. When the agent pursues an incorrect path, it often attempts minor, futile edits rather than restarting from a known good state. Additionally, it can miss obscure edge cases not explicitly detailed in natural language descriptions.

Future work will focus on incorporating test amplification and mutation testing to improve patch robustness. The authors also envision expanding the action space to include deployment, code review, and A/B testing, further evolving USEagent toward a fully autonomous AI software engineer.

Summary updated

Abstract

The growth of Large Language Model (LLM) technology has raised expectations for automated coding. However, software engineering is more than coding and is concerned with activities including maintenance and evolution of a project. In this context, the concept of LLM agents has gained traction, which utilize LLMs as reasoning engines to invoke external tools autonomously. But is an LLM agent the same as an AI software engineer? In this paper, we seek to understand this question by developing a Unified Software Engineering agent or USEagent. Unlike existing work which builds specialized agents for specific software tasks such as testing, debugging, and repair, our goal is to build a unified agent which can orchestrate and handle multiple capabilities. This gives the agent the promise of handling complex scenarios in software development such as fixing an incomplete patch, adding new features, or taking over code written by others. We envision USEagent as the first draft of a future AI Software Engineer which can be a team member in future software development teams involving both AI and humans. To evaluate the efficacy of USEagent, we build a Unified Software Engineering bench (USEbench) comprising of myriad tasks such as coding, testing, and patching. USEbench is a judicious mixture of tasks from existing benchmarks such as SWE-bench, SWT-bench, and REPOCOD. In an evaluation on USEbench consisting of 1,271 repository-level software engineering tasks, USEagent shows improved efficacy compared to existing general agents such as OpenHands CodeActAgent. There exist gaps in the capabilities of USEagent for certain coding tasks, which provides hints on further developing the AI Software Engineer of the future.
BibTeX
@misc{jiang2025unified,
  author = {Nan Jiang and Leonhard Applis and Yuntong Zhang and Shanchao Liang and Nan Jiang and Lin Tan and Abhik Roychoudhury},
  title = {{Unified Software Engineering agent as AI Software Engineer}},
  year = {2025},
  eprint = {2506.14683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.14683},
}
Figure