Abstract
Large Language Models (LLMs) have shown remarkable potential in code generation, yet their reliability is often limited by the quality of self-generated verification tools. We present Multi-Perspective Self-Consistency (MPSC), a framework that treats code solutions, specifications, and test cases as distinct, equally fallible perspectives. By modeling these perspectives in a tripartite graph and applying iterative manifold ranking, MPSC significantly improves code generation accuracy. Our evaluation shows that MPSC enables models like ChatGPT to surpass GPT-4 on major coding benchmarks, including HumanEval and CodeContests.
Introduction
While LLMs are increasingly capable of generating functional code, they frequently fail on complex logic or edge cases. A common mitigation strategy involves sampling multiple solutions and selecting the best one using a verifier—typically unit tests generated by the same model. However, this "judge-based" approach assumes the verifier is more reliable than the solution. In practice, LLM-generated tests are often as buggy as the code they evaluate. MPSC addresses this by moving toward a "consensus-based" model, where multiple reasoning outputs are cross-referenced to identify the most stable and correct result.
Methodology: The MPSC Framework
The MPSC framework operates through a structured three-stage pipeline designed to leverage the collective intelligence of diverse model outputs.
- Multi-Perspective Generation: Instead of sampling only code solutions, the LLM is prompted to generate three distinct perspectives:
- Solutions: The executable source code.
- Specifications: Pre-conditions and post-conditions written in Python to describe expected behavior.
- Test Cases: Input/output pairs for functional verification.
- Consistency Graph Construction: These outputs are treated as nodes in a 3-partite graph. The framework measures Inter-consistency (e.g., does the code pass the test?) and Intra-consistency (lexical or semantic similarity between solutions).
- Output Reranking: Rather than simple majority voting, MPSC employs an iterative manifold ranking algorithm. This process propagates confidence through the graph, identifying the solution that maintains the strongest holistic agreement across all three perspectives.
Experimental Evaluation
The authors evaluated MPSC across four major Python benchmarks: HumanEval, HumanEval+ (which includes 80x more unit tests), MBPP, and CodeContests.
The results indicate that MPSC provides a substantial performance boost. For ChatGPT (GPT-3.5-Turbo), Pass@1 accuracy increased by 15.91% on HumanEval and 9.37% on CodeContests. Notably, ChatGPT equipped with MPSC outperformed vanilla GPT-4 across all benchmarks. For example, on the rigorous HumanEval+ dataset, MPSC-ChatGPT achieved 73.47% compared to GPT-4's 70.52%.
Key Insights
- Equality of Perspectives: By treating specifications and tests as equally fallible as the code, MPSC becomes robust to the noise inherent in self-generated verification.
- Density as a Proxy for Truth: A strong positive correlation was observed between the "edge density" in the consistency graph and the likelihood of a solution being correct.
- Model Agnostic: The framework is highly generalizable, providing consistent improvements for both proprietary models and open-source alternatives like Code Llama and DeepSeek Coder.
Conclusion and Future Work
Multi-Perspective Self-Consistency represents a significant step forward in LLM reliability for coding. By shifting the focus from individual verification to holistic consensus, MPSC mitigates the "noisy verifier" problem. While currently optimized for code—where execution provides a deterministic measure of agreement—future research will explore applying similar multi-perspective consistency measures to general reasoning and subjective natural language tasks.
Summary updated