CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection

Qingyu Zhang, Puzhuo Liu, Peng Di, Chenxiong QianarXiv 2025 · DOI · arXiv

Introduction

In software engineering, the commit message provides the essential context for code changes. However, these messages are frequently low-quality or inaccurate, leading to Message-Code Inconsistency (MCI). MCI poses systemic risks, including misleading code reviewers, polluting AI training datasets, and potentially masking security vulnerabilities. CodeFuse-CommitEval is the first dedicated benchmark designed to evaluate how effectively Large Language Models (LLMs) can identify these discrepancies.

The CodeFuse-CommitEval Dataset

The researchers developed a mutation-based pipeline to create the CodeFuse-CommitEval dataset, leveraging high-quality commits from the ApacheCM repository. Using Claude Sonnet 4, they introduced seven types of mutations to create "inconsistent" examples:

  • Identifier Mutations: Swapping names for functions, files, or components.
  • Operation Mutations: Changing the nature of the change (e.g., "added" vs. "removed").
  • Purpose Mutations: Altering the stated intent while maintaining technical similarity.
  • Feature Mutations: Describing changes that did not occur or omitting significant ones.

A two-fold verification process ensured the reliability of the labels. The final dataset consists of 52,127 instances across 50 unique GitHub repositories, providing a diverse and verified foundation for benchmarking.

Methodology and Evaluation

The study benchmarked six state-of-the-art open-source LLMs, including DeepSeek-V3.1, Qwen3-30B, and gpt-oss-20B. Three prompting strategies were evaluated:

  • Vanilla: Standard zero-shot prompts.
  • Few-Shot: Providing contextual examples to the model.
  • Chain-of-Thought (CoT): Requiring the model to generate step-by-step reasoning.

Key Findings

The "Skepticism" Bias

LLMs generally demonstrate a "skepticism bias," where they are more effective at catching inconsistencies (Recall ~86%) than at confirming consistency (Specificity ~64%). This suggests that models are prone to flagging correct commits as errors, acting as overly cautious auditors.

Surface vs. Semantic Complexity

Surface-level inconsistencies, such as incorrect file paths or basic operation errors, are easily identified. However, "Purpose" mutations—where the underlying reason for a change is misrepresented—remain a significant challenge, requiring deeper code comprehension and reasoning.

The Impact of Context and Reasoning

While Chain-of-Thought (CoT) prompting improves precision, it comes with a high computational cost, often doubling token usage. Furthermore, providing additional code context (e.g., 20 lines of surrounding code) benefited larger models like gpt-oss-20B but introduced noise that degraded the performance of smaller models like Llama-3.1-8B.

Performance Results

The gpt-oss-20B model emerged as the top performer, achieving an accuracy of 85.61% and a Matthews Correlation Coefficient (MCC) of 0.6942. However, this performance came at the cost of high token usage. In contrast, Mistral-Small-3.2-24B demonstrated the highest token efficiency, making it a more viable candidate for high-volume auditing tasks despite slightly lower accuracy (~75%).

Conclusion and Future Work

CodeFuse-CommitEval highlights both the potential and the limitations of current LLMs in automated code auditing. While models are adept at spotting blatant errors, improving their grasp of high-level intent and reducing false-positive rates are critical next steps. Future research will explore optimal context lengths and more diverse reasoning paths to enhance detection capabilities for complex semantic inconsistencies.

Summary updated

Abstract

Version control relies on commit messages to convey the rationale for code changes, but these messages are often low quality and, more critically, inconsistent with their diffs-known as message-code inconsistency (MCI). MCIs mislead reviewers, hinder maintenance, contaminate research datasets, and may obscure security patches. Yet, no dedicated benchmark exists to evaluate models for MCI detection. We introduce CODEFUSE-COMMITEVAL, the first benchmark designed for MCI detection using large language models (LLMs). Built on the ApacheCM dataset for diversity and quality, we generate seven types of inconsistent messages through rule-guided mutations of originally consistent commits and apply two-fold validation to verify both positive and negative samples. Using this labeled dataset of message-diff pairs, we evaluate six state-of-the-art open-source LLMs under a vanilla setting and with three augmentation strategies: few-shot prompting, chain-of-thought, and extended context. Results show models detect inconsistent commits more reliably than consistent ones (average Recall 85.95%, Precision 80.28%, Specificity 63.8%); gpt-oss-20B performs best overall but uses over twice the tokens of others. Augmentation effects vary: adjacent context helps larger models but adds noise for smaller ones; few-shot improves accuracy and reduces token use, yet increases universally incorrect predictions; chain-of-thought boosts precision and specificity at the cost of recall and higher token consumption. Type-wise analysis reveals higher detectability for component, file-path, and operation inconsistencies, but lower accuracy and higher token cost for intent-level "purpose" inconsistencies. CODEFUSE-COMMITEVAL provides a rigorous foundation for measuring, comparing, and advancing MCI detection, highlighting the need for richer context and balanced data to capture high-level semantic gaps.
BibTeX
@misc{zhang2025codefuse,
  author = {Qingyu Zhang and Puzhuo Liu and Peng Di and Chenxiong Qian},
  title = {{CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection}},
  year = {2025},
  eprint = {2511.19875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.19875},
}
Figure