Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency

Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, Nan Duan

Abstract

Large Language Models (LLMs) have demonstrated significant potential in code generation, yet they often fail to produce correct solutions in a single attempt. Existing verification methods often rely on an "expert assumption," treating model-generated test cases as ground truth. This paper introduces Multi-Perspective Self-Consistency (MPSC), a framework that treats solutions, specifications, and test cases as equally noisy perspectives. By constructing a 3-partite graph and applying an iterative ranking algorithm, MPSC effectively "triangulates" the most reliable solution. Experimental results show that MPSC significantly boosts performance, enabling ChatGPT to surpass GPT-4 on major coding benchmarks like HumanEval.

Introduction

While LLMs exhibit impressive zero-shot coding capabilities, their outputs are inherently stochastic and prone to errors. To improve reliability, common strategies involve sampling multiple solutions and re-ranking them using verification properties, such as model-generated unit tests. However, these verifiers are often generated by the same model and are thus subject to similar noise and inaccuracies. If a test case is flawed, it may penalize a correct solution or validate an incorrect one. MPSC addresses this by moving away from the "expert assumption" and instead aggregating noisy, multi-modal outputs to identify the most consistent answer.

Methodology: The MPSC Framework

The MPSC framework operates through a structured three-stage process designed to evaluate consistency across different representations of a programming task.

The three-stage MPSC framework pipeline: Multi-Perspective Generation, Consistency Measures, and Output Reranking.

1. Multi-Perspective Generation

The LLM is prompted to generate three distinct types of outputs for a given problem:

  • Solutions: The actual source code implementation.
  • Specifications: Formal properties, such as pre-conditions and post-conditions, written in code.
  • Test Cases: Input-output pairs used for functional verification.

2. Graph Construction and Consistency Measures

These outputs are modeled as vertices in a 3-partite graph. Edges represent "inter-consistency" between different types (e.g., a solution passing a specific test case). The framework also evaluates "intra-consistency" within types. Notably, MPSC utilizes Semantic Intra-consistency, where two solutions are considered similar if they share the same functional behavior (i.e., they satisfy the same set of specifications and test cases), rather than just looking at lexical similarity.

3. Output Reranking

Finally, an iterative manifold ranking algorithm, conceptually similar to PageRank, propagates confidence scores across the graph. This process identifies the solution that maintains the highest functional consistency across all generated perspectives.

Key Insights

  • Equality of Perspectives: By treating test cases and specifications as noisy rather than absolute, the framework can triangulate the truth. High confidence is assigned to solutions that align with multiple, mutually consistent verifiers.
  • The Triangulation Effect: Performance gains are strongly correlated with edge density in the graph. As agreement between perspectives increases, the likelihood of selecting a correct solution grows exponentially.

Experimental Results

The framework was evaluated on four major Python benchmarks: HumanEval, HumanEval+, MBPP, and CodeContests.

Performance comparison of MPSC against baseline models across four coding benchmarks.

Using ChatGPT (GPT-3.5-Turbo) as the base model, MPSC achieved substantial improvements:

  • HumanEval: Pass@1 increased by +15.91%, reaching 84.29% and outperforming the raw performance of GPT-4 (81.48%).
  • HumanEval+: Improved Pass@1 by +15.64%, demonstrating robustness against augmented test suites.
  • CodeContests: Achieved a +9.37% improvement on highly challenging competitive programming tasks.

The iterative ranking process is highly efficient, typically converging in less than 0.1 seconds, making it a practical addition to LLM inference pipelines.

Conclusion and Future Work

MPSC demonstrates that treating various model outputs as complementary perspectives can significantly mitigate the noise inherent in LLM-generated code and verifiers. While highly effective on standard benchmarks, future research will focus on scaling this approach to more complex, real-world software environments where formal specifications are harder to generate. Additionally, exploring task-agnostic consistency measures could extend the benefits of MPSC to non-coding domains like mathematical reasoning and general question answering.

Summary updated

BibTeX
@inproceedings{huang2024enhancing,
  author = {Baizhou Huang and Shuai Lu and Weizhu Chen and Xiaojun Wan and Nan Duan},
  title = {{Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.78},
}
Figure