MetaGPT: Meta Programming for Multi-Agent Collaborative Framework

Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu

Introduction

While Large Language Models (LLMs) excel at generating isolated code snippets, they often struggle with the complexities of multi-file software engineering. Existing multi-agent frameworks frequently rely on unstructured natural language "chatting," which can lead to cascading hallucinations—where a single error snowballs into system failure—and unproductive "idle chatter." MetaGPT addresses these challenges by treating LLMs as a specialized software company, encoding human-like Standardized Operating Procedures (SOPs) into their collaborative workflows. By replacing open-ended dialogue with structured meta-programming, it achieves state-of-the-art performance on coding benchmarks.

Methodology: Standardized Operating Procedures (SOPs)

The core innovation of MetaGPT is the transition from open-ended dialogue to structured meta-programming. The framework implements a rigorous assembly-line workflow governed by SOPs, mimicking professional software engineering standards.

MetaGPT workflow illustrating role specialization and the transition from human requirements to executable code through structured SOPs.

Role Specialization and Structured Communication

MetaGPT defines five distinct roles: Product Manager, Architect, Project Manager, Engineer, and QA Engineer. Each role is governed by a specific profile, goal, and set of constraints. Instead of passing raw text, agents produce structured artifacts such as Product Requirement Documents (PRDs), system interface designs, and sequence diagrams.

To manage information flow and avoid overload, MetaGPT utilizes a Publish-Subscribe Mechanism. A shared message pool allows agents to subscribe only to relevant information, reducing noise. Furthermore, the framework incorporates Iterative Programming with Executable Feedback, where Engineer agents execute code, capture runtime errors, and refine solutions through an automated debugging loop (up to three retries).

Evaluation Benchmarks

The framework was evaluated across three key datasets:

  • HumanEval: 164 handwritten programming tasks focusing on functional correctness.
  • MBPP (Mostly Basic Python Problems): 427 Python tasks covering core concepts and standard library features.
  • SoftwareDev: A new dataset of 70 representative software development tasks (e.g., creating a "2048" game or a weather dashboard). This dataset tests "engineering" capabilities—handling multiple files and dependencies—rather than just algorithmic logic.

Key Results and Performance

MetaGPT significantly outperformed previous state-of-the-art (SoTA) models and frameworks across major coding benchmarks.

Comparison of Pass@1 scores on MBPP and HumanEval benchmarks, showing MetaGPT's superior performance compared to other models.

On HumanEval, MetaGPT achieved an 85.9% Pass@1, a massive jump over GPT-4’s standalone performance of 67.0%. On MBPP, it reached 87.7%. In the more complex SoftwareDev tasks, the framework achieved a 100% task completion rate. Beyond accuracy, MetaGPT demonstrated high efficiency, requiring approximately 124 tokens to generate one line of code, whereas competing frameworks like ChatDev required roughly 249 tokens.

Insights and Future Directions

The success of MetaGPT highlights that LLMs perform significantly better when constrained by professional standards. SOPs act as a governance layer that keeps hallucinations in check, while structured handoffs via documents prove more reliable than messy chat histories.

However, limitations remain. Currently, MetaGPT treats every project as a fresh start, lacking the "long-term memory" to learn from past mistakes across different projects. The system also faces challenges with complex UI/UX design and multimodal inputs. Future work will focus on "Human-in-the-loop" integration, allowing for mid-process feedback, and the development of an "AgentStore" for trading specialized agents with different skill sets.

Summary updated

Abstract

Remarkable progress has been made on automated problem solving through societies of agents based on large language models (LLMs). Existing LLM-based multi-agent systems can already solve simple dialogue tasks. Solutions to more complex tasks, however, are complicated through logic inconsistencies due to cascading hallucinations caused by naively chaining LLMs. Here we introduce MetaGPT, an innovative meta-programming framework incorporating efficient human workflows into LLM-based multi-agent collaborations. MetaGPT encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. MetaGPT utilizes an assembly line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks into subtasks involving many agents working together. On collaborative software engineering benchmarks, MetaGPT generates more coherent solutions than previous chat-based multi-agent systems. Our project can be found at https://github.com/geekan/MetaGPT
BibTeX
@inproceedings{hong2024metagpt,
  author = {Sirui Hong and Xiawu Zheng and Jonathan Chen and Yuheng Cheng and Jinlin Wang and Ceyao Zhang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu},
  title = {{MetaGPT: Meta Programming for Multi-Agent Collaborative Framework}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=VtmBAGCN7o},
}
Figure