Abstract
Automated code generation using large language models (LLMs) has gained attention due to its efficiency and adaptability. However, real-world coding tasks or benchmarks like HumanEval and StudentEval often lack dedicated training datasets, challenging existing few-shot prompting approaches that rely on reference examples. Inspired by human metamemory—a cognitive process involving recall and evaluation—we present a novel framework (namely M^2WF) for improving LLMs' one-time code generation. This approach enables LLMs to autonomously generate, evaluate, and utilize synthetic examples to enhance reliability and performance. Unlike prior methods, it minimizes dependency on curated data and adapts flexibly to various coding scenarios. Our experiments demonstrate significant improvements in coding benchmarks, offering a scalable and robust solution for data-free environments.
Introduction
While Large Language Models (LLMs) exhibit impressive coding capabilities, their performance often hinges on the availability of high-quality reference examples, typically provided via few-shot prompting. In many practical scenarios, such curated libraries of similar code snippets are unavailable, creating a "data-free" challenge. Existing solutions like AceCoder rely on external retrieval from training sets, which is not always feasible. Other approaches, such as Analogical Prompting, encourage models to generate their own examples, but these are prone to hallucinations that can mislead the final output.
To address this, we introduce M^2WF (Metamemory Workflow), a framework that empowers LLMs to reliably leverage their internal latent knowledge. By mimicking the human cognitive process of metamemory—the ability to monitor and regulate one's own memory—M^2WF allows models to autonomously generate and filter internal reference examples to guide the coding process.
Theoretical Foundation and Methodology
The M^2WF framework is grounded in the psychological concept of metamemory, which involves the interaction between memory content and monitoring processes.
As shown in the cognitive model above, metamemory consists of three primary phases: Recall, Evaluation, and Regulation. M^2WF translates these into a four-stage computational pipeline:
- Recall: Instead of querying an external database, the LLM is prompted to "remember" $K$ related programming problems, providing their solutions and implementation steps from its internal knowledge.
- Evaluation: The model acts as a self-critic, assigning a confidence score (0-100) to each recalled example. Only the top $M$ most reliable examples are retained, filtering out potential hallucinations.
- Planning: Using the high-confidence examples as a foundation, the LLM synthesizes a step-by-step implementation plan (a "tutorial") specifically for the target problem.
- Guidance: Finally, the LLM generates the actual code, strictly following the self-generated plan.
Unlike retrieval-based methods like AceCoder (illustrated in the figure above), M^2WF does not require a pre-existing training set. It relies on the model's ability to validate its "memories" before using them to construct a logical strategy.
Experimental Evaluation
We evaluated M^2WF across five diverse benchmarks to ensure its robustness across different difficulty levels and programming languages:
- HumanEval & HumanEval+: Standard Python code generation benchmarks.
- StudentEval: Real-world prompts written by students, reflecting messy human requirements.
- Codeforces: High-difficulty competitive programming problems.
- MultiPL-E: A multi-language benchmark covering C++, Java, PHP, and more.
Key Results
The experimental results across various models (GPT-4, ChatGPT, DeepSeek-Coder-V2, Mistral-7B) demonstrate the framework's effectiveness:
- Performance Gains: ChatGPT (GPT-3.5) showed a 29.43% improvement in average score on StudentEval compared to standard prompting. Mistral-7B's Pass@1 score on HumanEval rose from 26.12% to 32.32% with the full pipeline.
- Competitive Superiority: On Codeforces, GPT-4 with M^2WF achieved a score of 29.50, outperforming AceCoder (27.50) despite the latter using a 3-shot retrieval set.
- Cross-Language Robustness: Consistent gains were observed across all six tested languages in MultiPL-E, including a 3.70% boost in C++.
Discussion and Conclusion
The success of M^2WF highlights that self-correction and planning are critical for reliable code generation. The "Evaluation" stage acts as a vital filter against hallucinations, while the "Planning" stage bridges the gap between recalled knowledge and the target problem. Notably, M^2WF demonstrates that an LLM's internal knowledge can be more powerful than external retrieval when properly regulated.
However, the framework introduces increased token costs due to its multi-stage nature, leading to higher latency and API expenses. Future work will focus on optimizing token usage and extending this metamemory approach to integrated development environments (IDEs) for real-world software engineering tasks.
Summary updated