Abstract
While Large Language Models (LLMs) have become the predominant paradigm for automated code generation, current single-model approaches fundamentally ignore the heterogeneous computational strengths that different models exhibit across programming languages, algorithmic domains, and development stages. This paper challenges the single-model convention by introducing a multi-stage, performance-guided orchestration framework that dynamically routes coding tasks to the most suitable LLMs within a structured generate-fix-refine workflow. Our approach is grounded in a comprehensive empirical study of 17 state-of-the-art LLMs across five programming languages (Python, Java, C++, Go, and Rust) using HumanEval-X benchmark. The study, which evaluates both functional correctness and runtime performance metrics (execution time, mean/max memory utilization, and CPU efficiency), reveals pronounced performance heterogeneity by language, development stage, and problem category. Guided by these empirical insights, we present PerfOrch, an LLM agent that orchestrates top-performing LLMs for each task context through stage-wise validation and rollback mechanisms. Without requiring model fine-tuning, PerfOrch achieves substantial improvements over strong single-model baselines: average correctness rates of 96.22% and 91.37% on HumanEval-X and EffiBench-X respectively, surpassing GPT-4o's 78.66% and 49.11%. Beyond correctness gains, the framework delivers consistent performance optimizations, improving execution time for 58.76% of problems with median speedups ranging from 17.67% to 27.66% across languages on two benchmarks. The framework's plug-and-play architecture ensures practical scalability, allowing new LLMs to be profiled and integrated seamlessly, thereby offering a paradigm for production-grade automated software engineering that adapts to the rapidly evolving generative AI landscape.
Introduction
The current landscape of automated code generation is dominated by a "hero model" paradigm, where a single large language model (LLM) is expected to handle all aspects of software development. However, this approach overlooks the inherent heterogeneity in model capabilities. A model that excels in Python logic might struggle with the memory safety nuances of Rust or the complex syntax of C++. Furthermore, existing benchmarks primarily focus on functional correctness, neglecting the critical production requirements of runtime efficiency, such as latency and memory utilization.
This paper introduces PerfOrch, an orchestration framework designed to move beyond the single-model limitation. By treating different LLMs as a collaborative team—each acting as a specialized creator, debugger, or optimizer—PerfOrch dynamically routes tasks through a multi-stage pipeline to produce code that is both correct and performant.
The PerfOrch Framework
PerfOrch is built upon a "plug-and-play" architecture that leverages empirical performance data from 17 state-of-the-art LLMs, including GPT-4o, Claude 3.7 Sonnet, and Grok 3. The framework operates through three distinct modules:
- Generate Module: Initial coding tasks are routed to the model with the highest historical performance for the specific language and problem category (e.g., "Math" in Java).
- Fix Module: If the generated code fails unit tests, it enters an iterative debugging loop where specialized "fixer" models attempt to resolve the bugs.
- Refine Module: Once functional correctness is achieved, the code is passed to a "refiner" model to optimize for execution time, memory, or CPU efficiency.
The system employs a Sequential Acceptance strategy, querying ranked models one by one to balance optimization gains with computational costs. This allows the system to integrate new models seamlessly as they are released.
Experimental Results
Evaluation on the HumanEval-X and EffiBench-X benchmarks demonstrates that PerfOrch significantly outperforms single-model baselines. While GPT-4o achieves pass rates of 78.66% and 49.11% on these benchmarks, PerfOrch reaches near-perfect correctness with 96.22% and 91.37%, respectively.
Beyond correctness, the refinement stage provides substantial runtime benefits. The framework improved execution time for 58.76% of problems, with median speedups between 17.67% and 27.66%. These gains often stem from algorithmic optimizations, such as replacing $O(n^2)$ loops with $O(n)$ logic, which single-pass generation frequently misses. Additionally, the orchestration approach proved cost-effective, achieving a 22% reduction in API costs compared to exhaustive "Best-of-N" sampling strategies.
Conclusion
PerfOrch demonstrates that the future of automated software engineering lies in the intelligent orchestration of specialized models rather than the pursuit of a single universal LLM. While currently focused on function-level generation, the framework's modular design allows for future expansion into repository-level tasks and real-time developer collaboration. As the LLM landscape continues to evolve, this performance-guided orchestration provides a scalable path toward production-grade AI-driven development.
Summary updated