Abstract
This paper introduces CodeBERT-nt, a novel approach that leverages pre-trained transformer models to quantify "code naturalness" for identifying potential software bugs. By measuring the predictability of code tokens using CodeBERT, the authors develop a cross-project bug-ranking tool that overcomes the project-specific training requirements and generalization limitations of traditional n-gram models. Evaluation on the SmartShark dataset, comprising 2,510 buggy versions across 40 Java projects, demonstrates that CodeBERT-nt significantly outperforms complexity-based and random baselines. Notably, the model's prediction confidence serves as a superior indicator of bugginess compared to prediction accuracy.
Introduction
In software engineering, the Naturalness Hypothesis posits that source code, much like human language, is highly repetitive and predictable. Historically, researchers have employed statistical $n$-gram models to measure this naturalness, under the premise that "unnatural" or unpredictable lines of code are more likely to harbor bugs or code smells. However, $n$-gram models suffer from significant drawbacks: they require tedious project-specific training, are highly sensitive to hyperparameters, and often fail to generalize across different coding styles.
To address these challenges, the authors propose CodeBERT-nt. By utilizing CodeBERT—a large-scale model pre-trained on millions of projects—the approach captures general programming patterns that are applicable across diverse software repositories without the need for project-specific retraining.
Methodology
CodeBERT-nt treats code naturalness as a predictability metric. The workflow involves parsing Java source code into an Abstract Syntax Tree (AST) and systematically masking tokens to evaluate the model's ability to predict them.
The researchers evaluated three primary metrics derived from CodeBERT's output:
- CBnt_conf (Confidence): The probability assigned to the top prediction.
- CBnt_cos (Cosine Similarity): The similarity between the embeddings of the original and predicted tokens.
- CBnt_acc (Accuracy): A binary check of whether the top prediction matches the original token.
To aggregate these token-level scores into line-level rankings, various methods (min, max, mean, median, and entropy) were tested. The study utilized the SmartShark dataset, focusing on refined "business logic" lines to reduce noise from boilerplate code like imports and declarations.
Key Results and Insights
Experimental results indicate that CodeBERT-nt provides a cleaner and more effective signal for bug localization than traditional baselines. A key finding was that model confidence (CBnt_conf) is the most potent indicator; when the model is "confused" by a line of code, that line is statistically more likely to be buggy. Interestingly, this confidence signal was more useful than whether the model's prediction was actually correct.
As shown in the comparative analysis, CodeBERT-nt (specifically using the CBnt_conf metric) outperformed both uniform-random ranking and complexity-based ranking. It improved the "first hit" rank—the rank of the first buggy line identified—by 11% compared to random selection. Furthermore, CodeBERT-nt yielded results comparable to, and in some cases slightly better than, $n$-gram models that were specifically trained on the target projects, highlighting the power of pre-training for cross-project generalization.
Discussion and Limitations
While CodeBERT-nt shows great promise, several limitations remain. The current implementation is optimized for Java, requiring adaptation of AST parsing for other languages. Additionally, running large transformer models is computationally more intensive than simple $n$-gram lookups. Finally, "natural bugs"—logic errors that use syntactically correct but semantically wrong variables—remain difficult to detect through naturalness alone, as they may still appear predictable to the model.
Conclusion
CodeBERT-nt demonstrates that large-scale pre-trained models can effectively quantify code naturalness for bug localization. By providing a flexible, cross-project solution that outperforms traditional baselines, it offers a robust alternative to project-specific statistical models. Future work will explore multi-metric combinations and expansion to other programming languages.
Summary updated