Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study

Shuo Liu, Jacky Keung, Zhen Yang, Fang Liu, Qilin Zhou, Yihan Liao

TL;DR

This paper investigates whether Parameter-Efficient Fine-Tuning (PEFT) can handle "dynamic" code tasks—like predicting defects in a commit or generating commit messages—as effectively as it handles static code analysis. The researchers found that PEFT methods like Adapter Tuning and LoRA not only match or beat full-model fine-tuning in performance but do so with significantly lower computational costs and better results in low-resource or cross-lingual scenarios.

What Problem Does This Paper Address?

Pre-trained Language Models (PLMs) like CodeBERT and CodeT5 have revolutionized software engineering tasks. Traditionally, adapting these models to specific tasks requires Full-Model Fine-Tuning (FMFT), which updates every single parameter. This is computationally expensive and can lead to "catastrophic forgetting," where the model loses the general knowledge it gained during pre-training.

While Parameter-Efficient Fine-Tuning (PEFT) has shown promise in static tasks (like code search), the authors identify a critical gap: dynamic code changes. Most PLMs are pre-trained on static code snippets, but many real-world tasks involve "code diffs"—comparing a version of code before and after a change. It wasn't clear if tuning only 1-5% of a model's parameters (the PEFT approach) would be enough to bridge the gap between static pre-training and dynamic downstream tasks.

Examples of static vs. dynamic code comprehension. Static involves a single snippet/comment pair; dynamic involves a code diff and a commit message.

Methods

The authors compared FMFT against two popular PEFT techniques across five different PLMs (CodeBERT, GraphCodeBERT, PLBART, UniXcoder, and CodeT5):

  1. Adapter Tuning (AT): This method inserts small, trainable "bottleneck" layers (adapters) between the existing layers of the PLM. During training, the original model weights are frozen, and only these new adapters are updated.
  2. Low-Rank Adaptation (LoRA): Instead of adding new layers, LoRA injects trainable low-rank matrices into the model's attention layers to approximate the weight updates.
Architectural comparison between Full-Model Fine-Tuning, Adapter Tuning, and LoRA.

To understand why these methods work, the authors also designed three probing tasks:

  • Invalid Type Detection (TYP): Tests static semantic understanding.
  • Code Change Match (CCM): A global dynamic task—can the model match a diff to its commit message?
  • Line Type Prediction (LTP): A local dynamic task—can the model predict if a line was added, deleted, or kept?

Datasets

The study utilized several large-scale datasets:

  • JIT-Defects4J: Used for Just-In-Time Defect Prediction (JIT-DP).
  • MCMD (Multi-language Commit Message Dataset): Used for Commit Message Generation (CMG), covering Java, C#, C++, Python, and JavaScript.
  • FIRA: A dataset of commits from top GitHub projects used to construct the new dynamic probing tasks (CCM and LTP).

Key Insights and Ideas

  • Efficiency without Sacrifice: PEFT achieves state-of-the-art results in defect prediction while updating only a fraction of the parameters. In the JIT-DP task, Adapter Tuning and LoRA improved F1 scores by 8.39% and 9.87% respectively compared to previous SOTA baselines.
  • The "Catastrophic Forgetting" Shield: Because PEFT keeps the original weights frozen, it retains the general-purpose programming knowledge from pre-training better than full fine-tuning. This makes it particularly robust in cross-lingual scenarios (training on one language, testing on another).
  • Low-Resource Champion: When training data is scarce (e.g., only 1,000 samples), PEFT often outperforms FMFT. The authors suggest that FMFT overfits too easily on small datasets, whereas the constrained parameter space of PEFT acts as a natural regularizer.
  • Dynamic Understanding: Probing tasks revealed that PEFT actually helps models learn dynamic code semantics (how code changes over time) better than full fine-tuning, especially in the middle layers of the transformer architecture.

Key Results

  • Performance: In Commit Message Generation, PEFT performed comparably to FMFT (BLEU scores around 21-22), but with much higher efficiency.
  • Resource Savings: PEFT saved approximately 15-20 minutes per training epoch and reduced GPU memory consumption by 4,000-5,000 MB.
  • Cross-Lingual Gains: In cross-lingual commit message generation, PEFT methods showed an average performance boost of up to 7.56% over FMFT.

Limitations and Future Work

The authors acknowledge a few constraints:

  • Model Scale: The study focused on models up to 220M parameters (CodeT5). The impact of PEFT on much larger models (LLMs) in this specific domain remains to be explored.
  • Task Variety: While JIT-DP and CMG are representative, other tasks like automated patch correctness or bug fixing could provide further insights.
  • Hyperparameter Sensitivity: Finding the "optimal" rank for LoRA or bottleneck dimension for Adapters is still a manual, trial-and-error process that needs more automated solutions.

Future work aims to explore more complex probing tasks involving code structure and syntax, as well as testing these techniques on even larger pre-trained models.

Summary updated

BibTeX
@inproceedings{liu2024delving,
  author = {Shuo Liu and Jacky Keung and Zhen Yang and Fang Liu and Qilin Zhou and Yihan Liao},
  title = {{Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study}},
  booktitle = {Saner},
  year = {2024},
  doi = {10.1109/saner60148.2024.00055},
}

Loading related content...

Figure