Introduction
In software engineering, the commit message provides the essential context for code changes. However, these messages are frequently low-quality or inaccurate, leading to Message-Code Inconsistency (MCI). MCI poses systemic risks, including misleading code reviewers, polluting AI training datasets, and potentially masking security vulnerabilities. CodeFuse-CommitEval is the first dedicated benchmark designed to evaluate how effectively Large Language Models (LLMs) can identify these discrepancies.
The CodeFuse-CommitEval Dataset
The researchers developed a mutation-based pipeline to create the CodeFuse-CommitEval dataset, leveraging high-quality commits from the ApacheCM repository. Using Claude Sonnet 4, they introduced seven types of mutations to create "inconsistent" examples:
- Identifier Mutations: Swapping names for functions, files, or components.
- Operation Mutations: Changing the nature of the change (e.g., "added" vs. "removed").
- Purpose Mutations: Altering the stated intent while maintaining technical similarity.
- Feature Mutations: Describing changes that did not occur or omitting significant ones.
A two-fold verification process ensured the reliability of the labels. The final dataset consists of 52,127 instances across 50 unique GitHub repositories, providing a diverse and verified foundation for benchmarking.
Methodology and Evaluation
The study benchmarked six state-of-the-art open-source LLMs, including DeepSeek-V3.1, Qwen3-30B, and gpt-oss-20B. Three prompting strategies were evaluated:
- Vanilla: Standard zero-shot prompts.
- Few-Shot: Providing contextual examples to the model.
- Chain-of-Thought (CoT): Requiring the model to generate step-by-step reasoning.
Key Findings
The "Skepticism" Bias
LLMs generally demonstrate a "skepticism bias," where they are more effective at catching inconsistencies (Recall ~86%) than at confirming consistency (Specificity ~64%). This suggests that models are prone to flagging correct commits as errors, acting as overly cautious auditors.
Surface vs. Semantic Complexity
Surface-level inconsistencies, such as incorrect file paths or basic operation errors, are easily identified. However, "Purpose" mutations—where the underlying reason for a change is misrepresented—remain a significant challenge, requiring deeper code comprehension and reasoning.
The Impact of Context and Reasoning
While Chain-of-Thought (CoT) prompting improves precision, it comes with a high computational cost, often doubling token usage. Furthermore, providing additional code context (e.g., 20 lines of surrounding code) benefited larger models like gpt-oss-20B but introduced noise that degraded the performance of smaller models like Llama-3.1-8B.
Performance Results
The gpt-oss-20B model emerged as the top performer, achieving an accuracy of 85.61% and a Matthews Correlation Coefficient (MCC) of 0.6942. However, this performance came at the cost of high token usage. In contrast, Mistral-Small-3.2-24B demonstrated the highest token efficiency, making it a more viable candidate for high-volume auditing tasks despite slightly lower accuracy (~75%).
Conclusion and Future Work
CodeFuse-CommitEval highlights both the potential and the limitations of current LLMs in automated code auditing. While models are adept at spotting blatant errors, improving their grasp of high-level intent and reducing false-positive rates are critical next steps. Future research will explore optimal context lengths and more diverse reasoning paths to enhance detection capabilities for complex semantic inconsistencies.
Summary updated