Introduction
Pre-trained Language Models (PLMs) like CodeBERT and CodeT5 have revolutionized software engineering by enabling automated code analysis and generation. Traditionally, adapting these models to specific tasks requires Full-Model Fine-Tuning (FMFT), which updates every parameter. However, FMFT is computationally expensive and can lead to "catastrophic forgetting," where the model loses the general knowledge it gained during pre-training.
While Parameter-Efficient Fine-Tuning (PEFT) has shown promise in static tasks like code search, a critical gap remains: dynamic code changes. Most PLMs are pre-trained on static code snippets, but many real-world tasks involve "code diffs"—comparing versions of code before and after a change. This study investigates whether tuning only 1-5% of a model's parameters can effectively bridge the gap between static pre-training and dynamic downstream tasks.
Methodology
The researchers compared FMFT against two popular PEFT techniques across five different PLMs (CodeBERT, GraphCodeBERT, PLBART, UniXcoder, and CodeT5):
- Adapter Tuning (AT): This method inserts small, trainable "bottleneck" layers (adapters) between the existing layers of the PLM. During training, the original model weights are frozen, and only these new adapters are updated.
- Low-Rank Adaptation (LoRA): Instead of adding new layers, LoRA injects trainable low-rank matrices into the model's attention layers to approximate the weight updates.
To understand why these methods work, the authors also designed three probing tasks:
- Invalid Type Detection (TYP): Tests static semantic understanding.
- Code Change Match (CCM): A global dynamic task—can the model match a diff to its commit message?
- Line Type Prediction (LTP): A local dynamic task—can the model predict if a line was added, deleted, or kept?
Experimental Setup
The study utilized several large-scale datasets:
- JIT-Defects4J: Used for Just-In-Time Defect Prediction (JIT-DP).
- MCMD (Multi-language Commit Message Dataset): Used for Commit Message Generation (CMG), covering Java, C#, C++, Python, and JavaScript.
- FIRA: A dataset of commits from top GitHub projects used to construct the new dynamic probing tasks (CCM and LTP).
Key Insights and Results
Efficiency without Sacrifice
PEFT achieves state-of-the-art results in defect prediction while updating only a fraction of the parameters. In the JIT-DP task, Adapter Tuning and LoRA improved F1 scores by 8.39% and 9.87% respectively compared to previous SOTA baselines. In Commit Message Generation, PEFT performed comparably to FMFT (BLEU scores around 21-22), but with much higher efficiency.
The "Catastrophic Forgetting" Shield
Because PEFT keeps the original weights frozen, it retains the general-purpose programming knowledge from pre-training better than full fine-tuning. This makes it particularly robust in cross-lingual scenarios (training on one language, testing on another), showing an average performance boost of up to 7.56% over FMFT.
Low-Resource Champion
When training data is scarce (e.g., only 1,000 samples), PEFT often outperforms FMFT. The constrained parameter space of PEFT acts as a natural regularizer, preventing the overfitting common in FMFT on small datasets.
Resource Savings
PEFT saved approximately 15-20 minutes per training epoch and reduced GPU memory consumption by 4,000-5,000 MB compared to FMFT.
Conclusion
Probing tasks revealed that PEFT helps models learn dynamic code semantics better than full fine-tuning, especially in the middle layers of the transformer architecture. While the study focused on models up to 220M parameters, the results strongly suggest that PEFT is a superior strategy for adapting PLMs to the complexities of code change learning. Future work will explore these techniques on larger LLMs and more diverse software engineering tasks.
Summary updated