Abstract
Recent advancements in AI for software engineering have largely focused on specialized agents for specific tasks like bug fixing or test generation. This paper introduces USEagent, a unified AI agent designed as a generalist software engineer. By employing a central "Meta-Agent" to orchestrate modular actions, USEagent achieves a 33.3% success rate across the new USEbench—a massive 1,271-task benchmark. The agent matches the performance of specialized tools while handling a significantly broader range of engineering challenges through dynamic workflow construction.
Introduction
The current landscape of AI-driven software engineering is fragmented, characterized by "specialist" agents optimized for narrow tasks such as program repair or unit test generation. However, human software engineers are generalists who navigate projects, reproduce bugs, and implement features interchangeably. Existing agents often rely on rigid, fixed workflows that struggle with tasks outside their specific design.
To bridge this gap, researchers developed USEagent. The core research question asks: Can we build a unified agent that autonomously decides which tools to use and which workflows to construct on the fly?
Methods: The Meta-Agent Architecture
USEagent moves away from hardcoded pipelines toward an autonomous orchestration model. Its architecture is built upon three pillars:
- The Meta-Agent: A central LLM-based reasoning engine (e.g., Claude 3.5 Sonnet) operating in a ReAct-style loop. It reasons about the current state and selects the next action dynamically.
- Modular Actions: Encapsulated "units of work" including
CodeRetrieval,Reproduction,EditCode, andExecuteTests. - Task State (Consensus Memory): A structured shared memory tracking code locations, test results, and version diffs, ensuring continuity across different phases of a task.
By disassembling fixed workflows, USEagent self-configures its path based on the task at hand. For instance, it might prioritize reproduction for a bug fix but lead with code retrieval for a new feature.
USEbench: A Unified Benchmark
The authors introduce USEbench, a "meta-benchmark" comprising 1,271 repository-level tasks. It aggregates several existing specialized benchmarks into a single unified interface to test generalist capabilities.
The benchmark includes:
- SWE-bench (Verified): Real-world GitHub issues for program repair.
- SWT-bench: Regression testing tasks.
- REPOCOD & REPOTEST: Repository-level code and test generation.
- Compound Tasks: Complex scenarios like "Partial Fix" and "Feature Development."
Key Insights and Results
USEagent demonstrates that a generalist approach does not necessarily sacrifice efficacy. On the SWE-Verified benchmark, it achieved a 45.6% success rate, nearly matching the 46.2% of AutoCodeRover, a tool specifically optimized for that task.
Dynamic Workflows and Recovery
A critical finding was that dynamic workflows performed 4% to 33% better than static sequences. The Meta-Agent's ability to "reason" its way out of sub-optimal initial decisions—such as trying a different retrieval strategy if the first fails—is a significant advantage over rigid pipelines.
Performance Metrics
- Overall Efficacy: USEagent solved 33.3% of all USEbench tasks, significantly outperforming general-purpose agents like OpenHands CodeActAgent (26.8%).
- Task-Specific Success: The agent achieved 40.3% success in reproduction and 31.8% in test generation. However, full-method generation (REPOCOD) remains a challenge at 6.0%.
- Cost Efficiency: Using open-source models like DeepSeek-V3 reduced costs by approximately 85% while maintaining competitive performance.
Limitations and Future Directions
Despite its success, USEagent faces challenges with effective backtracking. When the agent pursues an incorrect path, it often attempts minor, futile edits rather than restarting from a known good state. Additionally, it can miss obscure edge cases not explicitly detailed in natural language descriptions.
Future work will focus on incorporating test amplification and mutation testing to improve patch robustness. The authors also envision expanding the action space to include deployment, code review, and A/B testing, further evolving USEagent toward a fully autonomous AI software engineer.
Summary updated