Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency

Baizhou Huang, Shuai Lu, Xiaojun Wan, Nan Duan

Abstract

Large Language Models (LLMs) have shown remarkable potential in code generation, yet their reliability is often limited by the quality of self-generated verification tools. We present Multi-Perspective Self-Consistency (MPSC), a framework that treats code solutions, specifications, and test cases as distinct, equally fallible perspectives. By modeling these perspectives in a tripartite graph and applying iterative manifold ranking, MPSC significantly improves code generation accuracy. Our evaluation shows that MPSC enables models like ChatGPT to surpass GPT-4 on major coding benchmarks, including HumanEval and CodeContests.

Introduction

While LLMs are increasingly capable of generating functional code, they frequently fail on complex logic or edge cases. A common mitigation strategy involves sampling multiple solutions and selecting the best one using a verifier—typically unit tests generated by the same model. However, this "judge-based" approach assumes the verifier is more reliable than the solution. In practice, LLM-generated tests are often as buggy as the code they evaluate. MPSC addresses this by moving toward a "consensus-based" model, where multiple reasoning outputs are cross-referenced to identify the most stable and correct result.

Methodology: The MPSC Framework

The MPSC framework operates through a structured three-stage pipeline designed to leverage the collective intelligence of diverse model outputs.

  1. Multi-Perspective Generation: Instead of sampling only code solutions, the LLM is prompted to generate three distinct perspectives:
  2. Solutions: The executable source code.
  3. Specifications: Pre-conditions and post-conditions written in Python to describe expected behavior.
  4. Test Cases: Input/output pairs for functional verification.
  5. Consistency Graph Construction: These outputs are treated as nodes in a 3-partite graph. The framework measures Inter-consistency (e.g., does the code pass the test?) and Intra-consistency (lexical or semantic similarity between solutions).
  6. Output Reranking: Rather than simple majority voting, MPSC employs an iterative manifold ranking algorithm. This process propagates confidence through the graph, identifying the solution that maintains the strongest holistic agreement across all three perspectives.
The MPSC framework pipeline: from multi-perspective generation to graph-based reranking.

Experimental Evaluation

The authors evaluated MPSC across four major Python benchmarks: HumanEval, HumanEval+ (which includes 80x more unit tests), MBPP, and CodeContests.

The results indicate that MPSC provides a substantial performance boost. For ChatGPT (GPT-3.5-Turbo), Pass@1 accuracy increased by 15.91% on HumanEval and 9.37% on CodeContests. Notably, ChatGPT equipped with MPSC outperformed vanilla GPT-4 across all benchmarks. For example, on the rigorous HumanEval+ dataset, MPSC-ChatGPT achieved 73.47% compared to GPT-4's 70.52%.

Radar chart comparing MPSC-enhanced ChatGPT against GPT-4 and other baselines across four coding benchmarks.

Key Insights

  • Equality of Perspectives: By treating specifications and tests as equally fallible as the code, MPSC becomes robust to the noise inherent in self-generated verification.
  • Density as a Proxy for Truth: A strong positive correlation was observed between the "edge density" in the consistency graph and the likelihood of a solution being correct.
  • Model Agnostic: The framework is highly generalizable, providing consistent improvements for both proprietary models and open-source alternatives like Code Llama and DeepSeek Coder.

Conclusion and Future Work

Multi-Perspective Self-Consistency represents a significant step forward in LLM reliability for coding. By shifting the focus from individual verification to holistic consensus, MPSC mitigates the "noisy verifier" problem. While currently optimized for code—where execution provides a deterministic measure of agreement—future research will explore applying similar multi-perspective consistency measures to general reasoning and subjective natural language tasks.

Summary updated

Abstract

Large language models (LLMs) have exhibited remarkable ability in textual generation. However, in complex reasoning tasks such as code generation, generating the correct answer in a single attempt remains a formidable challenge for LLMs. Previous research has explored solutions by aggregating multiple outputs, leveraging the consistency among them. However, none of them have comprehensively captured this consistency from different perspectives. In this paper, we propose the Multi-Perspective Self-Consistency (MPSC) framework, a novel decoding strategy for LLM that incorporates both inter-consistency across outputs from multiple perspectives and intra-consistency within a single perspective. Specifically, we ask LLMs to sample multiple diverse outputs from various perspectives for a given query and then construct a multipartite graph based on them. With two predefined measures of consistency, we embed both inter- and intra-consistency information into the graph. The optimal choice is then determined based on consistency analysis in the graph. We conduct comprehensive evaluation on the code generation task by introducing solution, specification and test case as three perspectives. We leverage a code interpreter to quantitatively measure the inter-consistency and propose several intra-consistency measure functions. Our MPSC framework significantly boosts the performance on various popular benchmarks, including HumanEval (+17.60%), HumanEval Plus (+17.61%), MBPP (+6.50%) and CodeContests (+11.82%) in Pass$@1$, when compared to original outputs generated from ChatGPT, and even surpassing GPT-4.
BibTeX
@inproceedings{huang2024enhancing,
  author = {Baizhou Huang and Shuai Lu and Xiaojun Wan and Nan Duan},
  title = {{Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.78},
}
Figure