Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study

Shuo Liu, Jacky Keung, Zhen Yang, Fang Liu, Qilin Zhou, Yihan Liao

Introduction

Pre-trained Language Models (PLMs) like CodeBERT and CodeT5 have revolutionized software engineering by enabling automated code analysis and generation. Traditionally, adapting these models to specific tasks requires Full-Model Fine-Tuning (FMFT), which updates every parameter. However, FMFT is computationally expensive and can lead to "catastrophic forgetting," where the model loses the general knowledge it gained during pre-training.

While Parameter-Efficient Fine-Tuning (PEFT) has shown promise in static tasks like code search, a critical gap remains: dynamic code changes. Most PLMs are pre-trained on static code snippets, but many real-world tasks involve "code diffs"—comparing versions of code before and after a change. This study investigates whether tuning only 1-5% of a model's parameters can effectively bridge the gap between static pre-training and dynamic downstream tasks.

Comparison of static code analysis versus dynamic code change comprehension. Static tasks involve single snippets, while dynamic tasks require understanding code diffs and their context.

Methodology

The researchers compared FMFT against two popular PEFT techniques across five different PLMs (CodeBERT, GraphCodeBERT, PLBART, UniXcoder, and CodeT5):

  1. Adapter Tuning (AT): This method inserts small, trainable "bottleneck" layers (adapters) between the existing layers of the PLM. During training, the original model weights are frozen, and only these new adapters are updated.
  2. Low-Rank Adaptation (LoRA): Instead of adding new layers, LoRA injects trainable low-rank matrices into the model's attention layers to approximate the weight updates.
Architectural overview of Low-Rank Adaptation (LoRA), showing how low-rank matrices are integrated with pre-trained weights to enable efficient fine-tuning.

To understand why these methods work, the authors also designed three probing tasks:

  • Invalid Type Detection (TYP): Tests static semantic understanding.
  • Code Change Match (CCM): A global dynamic task—can the model match a diff to its commit message?
  • Line Type Prediction (LTP): A local dynamic task—can the model predict if a line was added, deleted, or kept?

Experimental Setup

The study utilized several large-scale datasets:

  • JIT-Defects4J: Used for Just-In-Time Defect Prediction (JIT-DP).
  • MCMD (Multi-language Commit Message Dataset): Used for Commit Message Generation (CMG), covering Java, C#, C++, Python, and JavaScript.
  • FIRA: A dataset of commits from top GitHub projects used to construct the new dynamic probing tasks (CCM and LTP).

Key Insights and Results

Efficiency without Sacrifice

PEFT achieves state-of-the-art results in defect prediction while updating only a fraction of the parameters. In the JIT-DP task, Adapter Tuning and LoRA improved F1 scores by 8.39% and 9.87% respectively compared to previous SOTA baselines. In Commit Message Generation, PEFT performed comparably to FMFT (BLEU scores around 21-22), but with much higher efficiency.

The "Catastrophic Forgetting" Shield

Because PEFT keeps the original weights frozen, it retains the general-purpose programming knowledge from pre-training better than full fine-tuning. This makes it particularly robust in cross-lingual scenarios (training on one language, testing on another), showing an average performance boost of up to 7.56% over FMFT.

Low-Resource Champion

When training data is scarce (e.g., only 1,000 samples), PEFT often outperforms FMFT. The constrained parameter space of PEFT acts as a natural regularizer, preventing the overfitting common in FMFT on small datasets.

Resource Savings

PEFT saved approximately 15-20 minutes per training epoch and reduced GPU memory consumption by 4,000-5,000 MB compared to FMFT.

Conclusion

Probing tasks revealed that PEFT helps models learn dynamic code semantics better than full fine-tuning, especially in the middle layers of the transformer architecture. While the study focused on models up to 220M parameters, the results strongly suggest that PEFT is a superior strategy for adapting PLMs to the complexities of code change learning. Future work will explore these techniques on larger LLMs and more diverse software engineering tasks.

Summary updated

BibTeX
@inproceedings{liu2024delving,
  author = {Shuo Liu and Jacky Keung and Zhen Yang and Fang Liu and Qilin Zhou and Yihan Liao},
  title = {{Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study}},
  booktitle = {Saner},
  year = {2024},
  doi = {10.1109/saner60148.2024.00055},
}
Figure