CodeBERT-nt: code naturalness via CodeBERT

Ahmed Khanfir, Matthieu Jimenez, Mike Papadakis, Yves Le Traon

Abstract

This paper introduces CodeBERT-nt, a novel approach that leverages pre-trained transformer models to quantify "code naturalness" for identifying potential software bugs. By measuring the predictability of code tokens using CodeBERT, the authors develop a cross-project bug-ranking tool that overcomes the project-specific training requirements and generalization limitations of traditional n-gram models. Evaluation on the SmartShark dataset, comprising 2,510 buggy versions across 40 Java projects, demonstrates that CodeBERT-nt significantly outperforms complexity-based and random baselines. Notably, the model's prediction confidence serves as a superior indicator of bugginess compared to prediction accuracy.

Introduction

In software engineering, the Naturalness Hypothesis posits that source code, much like human language, is highly repetitive and predictable. Historically, researchers have employed statistical $n$-gram models to measure this naturalness, under the premise that "unnatural" or unpredictable lines of code are more likely to harbor bugs or code smells. However, $n$-gram models suffer from significant drawbacks: they require tedious project-specific training, are highly sensitive to hyperparameters, and often fail to generalize across different coding styles.

To address these challenges, the authors propose CodeBERT-nt. By utilizing CodeBERT—a large-scale model pre-trained on millions of projects—the approach captures general programming patterns that are applicable across diverse software repositories without the need for project-specific retraining.

Methodology

CodeBERT-nt treats code naturalness as a predictability metric. The workflow involves parsing Java source code into an Abstract Syntax Tree (AST) and systematically masking tokens to evaluate the model's ability to predict them.

Overview of the CodeBERT-nt pipeline.

The researchers evaluated three primary metrics derived from CodeBERT's output:

  • CBnt_conf (Confidence): The probability assigned to the top prediction.
  • CBnt_cos (Cosine Similarity): The similarity between the embeddings of the original and predicted tokens.
  • CBnt_acc (Accuracy): A binary check of whether the top prediction matches the original token.

To aggregate these token-level scores into line-level rankings, various methods (min, max, mean, median, and entropy) were tested. The study utilized the SmartShark dataset, focusing on refined "business logic" lines to reduce noise from boilerplate code like imports and declarations.

Key Results and Insights

Experimental results indicate that CodeBERT-nt provides a cleaner and more effective signal for bug localization than traditional baselines. A key finding was that model confidence (CBnt_conf) is the most potent indicator; when the model is "confused" by a line of code, that line is statistically more likely to be buggy. Interestingly, this confidence signal was more useful than whether the model's prediction was actually correct.

Comparison of ranking approaches for 1st hit and mean rank.

As shown in the comparative analysis, CodeBERT-nt (specifically using the CBnt_conf metric) outperformed both uniform-random ranking and complexity-based ranking. It improved the "first hit" rank—the rank of the first buggy line identified—by 11% compared to random selection. Furthermore, CodeBERT-nt yielded results comparable to, and in some cases slightly better than, $n$-gram models that were specifically trained on the target projects, highlighting the power of pre-training for cross-project generalization.

Discussion and Limitations

While CodeBERT-nt shows great promise, several limitations remain. The current implementation is optimized for Java, requiring adaptation of AST parsing for other languages. Additionally, running large transformer models is computationally more intensive than simple $n$-gram lookups. Finally, "natural bugs"—logic errors that use syntactically correct but semantically wrong variables—remain difficult to detect through naturalness alone, as they may still appear predictable to the model.

Conclusion

CodeBERT-nt demonstrates that large-scale pre-trained models can effectively quantify code naturalness for bug localization. By providing a flexible, cross-project solution that outperforms traditional baselines, it offers a robust alternative to project-specific statistical models. Future work will explore multi-metric combinations and expansion to other programming languages.

Summary updated

Abstract

Much of software-engineering research relies on the naturalness of code, the fact that code, in small code snippets, is repetitive and can be predicted using statistical language models like n-gram. Although powerful, training such models on large code corpus is tedious, time-consuming and sensitive to code patterns (and practices) encountered during training. Consequently, these models are often trained on a small corpora and estimate the language naturalness that is relative to a specific style of programming or type of project. To overcome these issues, we propose using pre-trained language models to infer code naturalness. Pre-trained models are often built on big data, are easy to use in an out-of-the-box way and include powerful learning associations mechanisms. Our key idea is to quantify code naturalness through its predictability, by using state-of-the-art generative pre-trained language models. To this end, we infer naturalness by masking (omitting) code tokens, one at a time, of code-sequences, and checking the models' ability to predict them. To this end, we evaluate three different predictability metrics; a) measuring the number of exact matches of the predictions, b) computing the embedding similarity between the original and predicted code, i.e., similarity at the vector space, and c) computing the confidence of the model when doing the token completion task irrespective of the outcome. We implement this workflow, named CodeBERT-nt, and evaluate its capability to prioritize buggy lines over non-buggy ones when ranking code based on its naturalness. Our results, on 2510 buggy versions of 40 projects from the SmartShark dataset, show that CodeBERT-nt outperforms both, random-uniform and complexity-based ranking techniques, and yields comparable results (slightly better) than the n-gram models.
BibTeX
@inproceedings{khanfir2022codebert,
  author = {Ahmed Khanfir and Matthieu Jimenez and Mike Papadakis and Yves Le Traon},
  title = {{CodeBERT-nt: code naturalness via CodeBERT}},
  booktitle = {QRS},
  year = {2022},
  url = {https://openreview.net/forum?id=uSM3ErcwXcW},
}
Figure