APPT: Boosting Automated Patch Correctness Prediction via Fine-tuning Pre-trained Models

Quanjun Zhang, Chunrong Fang, Weisong Sun, Yan Liu, Tieke He, Xiaodong Hao, Zhenyu Chen

Here is a concise, engaging summary of the paper designed for a technically literate audience.

TL;DR

Automated Program Repair (APR) tools often generate "plausible" patches that pass existing tests but fail in the real world due to overfitting. This paper introduces APPT, a novel patch correctness assessment technique that fine-tunes pre-trained language models (like BERT) end-to-end alongside an LSTM stack and classifier. By moving beyond static feature extraction to full pipeline domain adaptation, APPT sets a new state-of-the-art in distinguishing truly correct code fixes from deceptive, overfitted ones.

What Problem Does This Paper Address?

Automated Program Repair (APR) holds massive promise for reducing developer workload by automatically fixing software bugs. However, its biggest bottleneck is the overfitting problem: an APR tool generates a patch that passes all developer-written test cases (making it plausible), but it fails to capture the true intended semantics of the program, often breaking other unexercised functionalities.

To filter out these bad patches, researchers use Automated Patch Correctness Assessment (APCA) techniques. Recent trends have leaned toward using pre-trained language models like BERT to convert code into vector representations for classification. However, prior approaches treated these large models as frozen feature extractors. Because the model's weights weren't updated during training, the resulting code embeddings remained sub-optimal for the highly specific task of reasoning about patch correctness.

Methods

The authors propose APPT (Automated Pre-trained model-based Patch correcTness assessment), an end-to-end pipeline that fine-tunes the underlying language model specifically for patch classification. The pipeline consists of three main phases:

  1. Code Extraction: The buggy and patched code snippets are parsed out from the patch file, retaining surrounding context lines to preserve semantic meaning. The snippets are tokenized using subword tokenization.
  2. Token Embedding: Instead of keeping the encoder frozen, APPT passes the tokens through a pre-trained encoder stack (implemented using BERT, CodeBERT, or GraphCodeBERT) that is actively fine-tuned during training.
  3. Patch Classification: The embedding vectors for the buggy and patched code are integrated via concatenation. This joint vector is fed into a bidirectional LSTM stack to capture long-range token dependencies and sequential context. Finally, a deep learning classifier with fully connected layers and a softmax function outputs the probability of the patch being correct versus overfitted.
Fig. 2: Overview of APPT

Datasets

The authors evaluated APPT using two distinct patch datasets:

  • Small Dataset: Consists of 1,183 labeled patches generated by 22 distinct APR tools evaluated on the Defects4J benchmark. It contains 648 overfitting patches and 535 correct patches.
  • Large Dataset: A massive, balanced dataset containing 49,694 real-world labeled patches compiled from the ManySStuBs4J benchmark (providing correct fixes mined from open-source projects) and the RepairThemAll framework (providing overfitting patches).

Key Insights and Ideas

  • Fine-Tuning is the Secret Sauce: The biggest "aha" moment is that freezing large pre-trained models limits their utility in software engineering tasks. Fine-tuning the encoder end-to-end allows the model to undergo domain adaptation, shifting its generic language understanding into a highly specialized capability for detecting subtle code discrepancies.
  • Sequence Features Outperform Complex Structures: Surprisingly, treating code as a sequence of tokens (enhanced by an LSTM stack) outperformed more complex, structure-heavy approaches (like AST path extraction used in CACHE) or manually engineered features (used in ODS).
  • Model Generalizability: The framework is model-agnostic. Swapping the standard NLP-focused BERT for code-specific models like CodeBERT or GraphCodeBERT yielded immediate performance boosts, proving that APPT scales naturally with advancements in underlying LLMs.

Key Results

  • On the Small Dataset, APPT achieved an accuracy of 79.7% and a recall of 83.2%, outperforming the previous state-of-the-art technique, CACHE, by 4.3% and 6.7% respectively.
  • On the Large Dataset, APPT achieved near-perfect performance, exceeding 99% across all five primary evaluation metrics (Accuracy, Precision, Recall, F1-score, and AUC).
  • Ablation Studies proved the necessity of each component: removing the fine-tuning process dropped the F1-score by 10.22%, while removing the LSTM stack dropped it by 4.11% on the small dataset.
  • Upgrading the encoder to GraphCodeBERT improved APPT's precision by 2.8% and AUC by 3.3% compared to the base BERT implementation.

Limitations and Future Work

  • Cross-Project Performance Drop: Like most machine learning models, APPT experiences a performance dip when evaluated in a cross-project scenario (predicting correctness on entirely unseen software projects), dropping to around 72.49% accuracy.
  • Data Scale Sensitivity: The authors noted that more data isn't always better; selecting a highly informative, balanced training set is critical to maintaining high precision.
  • Future Directions: The authors suggest exploring the capabilities of even larger generative models (like ChatGPT) for the APCA task and investigating how structural features (like data flow graphs or abstract syntax trees) can be explicitly fused with APPT's sequential token pipeline.

Summary updated

Abstract

Automated program repair (APR) aims to fix software bugs automatically without human debugging efforts and plays a crucial role in software development and maintenance. Despite promising, APR is still challenged by a long-standing overfitting problem (i.e., the generated patch is plausible but overfitting). Various techniques have thus been proposed to address the overfitting problem. Recently, researchers have employed BERT to extract code features, which are then used to train a classifier for patch correctness prediction. However, BERT is restricted to feature extraction for classifier training without benefiting from the training process, potentially generating sub-optimal vector representations for patched code snippets. In this paper, we propose APPT, a pre-trained model-based automated patch correctness assessment technique by both pre-training and fine-tuning. APPT adopts a pre-trained model as the encoder stack, followed by an LSTM stack and a deep learning classifier. More importantly, the pre-trained model is fine-tuned in conjunction with other components as a whole pipeline to fully adapt it specifically for reasoning about patch correctness. We conduct an extensive experiment on 1,183 Defects4J patches and the experimental results show that APPT achieves prediction accuracy of 79.7% and recall of 83.2%, outperforming CACHE by 4.3% and 6.7%. Our additional investigation on 49,694 real-world patches shows that APPT achieves the optimum performance compared with existing representation learning techniques. We further investigate the impact of each component and find that they all positively contribute to APPT, e.g., the fine-tuning process and the LSTM stack increase F1-score by 10.22% and 4.11%, respectively. We also prove that adopting advanced pre-trained models can further provide substantial advancement, highlighting the generalizability of APPT.
BibTeX
@article{zhang2024appt,
  author = {Quanjun Zhang and Chunrong Fang and Weisong Sun and Yan Liu and Tieke He and Xiaodong Hao and Zhenyu Chen},
  title = {{APPT: Boosting Automated Patch Correctness Prediction via Fine-tuning Pre-trained Models}},
  journal = {IEEE Trans. Software Eng.},
  year = {2024},
  doi = {10.1109/tse.2024.3354969},
}
Figure