One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization

Deze Wang, Boxing Chen, Shanshan Li, Wei Luo, Shaoliang Peng, Wei Dong, Xiangke Liao

Abstract

While multilingual training is intended to facilitate knowledge sharing across programming languages, state-of-the-art models like CodeT5 and UniXcoder often suffer from performance degradation due to "catastrophic forgetting." This research introduces a parameter-efficient solution using adapter tuning. By freezing the backbone model and updating only 0.6% of the total parameters through specialized adapter modules, the researchers achieved new state-of-the-art results across six programming languages within a single, unified model.

Introduction: The Multilingual Paradox

In the field of AI for software engineering, the standard practice involves taking a massive pre-trained model and fine-tuning it for specific tasks, such as code summarization or semantic search. Traditionally, this required maintaining separate, full-sized model weights for every supported programming language—a significant burden for deployment and maintenance.

While multilingual fine-tuning (training one model on all languages simultaneously) worked for earlier models like CodeBERT, this study identifies a "multilingual paradox" in more advanced architectures like CodeT5 and UniXcoder. In these models, simultaneous training on multiple languages actually degrades performance compared to monolingual training. This occurs because the fine-tuning process overwrites critical general programming logic learned during pre-training, a phenomenon known as catastrophic forgetting.

Methodology: The Adapter Approach

To resolve the conflict between multilingual versatility and model performance, the authors propose Adapter Tuning. This method shifts the focus from updating the entire model to training small, modular components. The process follows three core steps:

  1. Freeze the Pre-trained Model: The original weights of the backbone (e.g., CodeT5) are locked, ensuring the foundational knowledge remains intact.
  2. Insert Adapters: Small, trainable "bottleneck" layers are inserted into each block of the Transformer architecture.
  3. Targeted Training: During the fine-tuning phase, only these tiny adapter layers (representing roughly 0.6% of the total parameters) are updated.

This architecture allows the frozen backbone to act as a permanent repository of general code logic, while the adapters learn the specific nuances of different tasks and programming languages.

Experimental Setup and Datasets

The researchers evaluated their approach using the CodeSearchNet and CodeXGLUE benchmarks, focusing on two primary tasks:

  • Code Search: Retrieving relevant code snippets based on natural language queries.
  • Code Summarization: Generating human-readable documentation from source code.

The study covered six major languages: Ruby, JavaScript, Java, Go, PHP, and Python. Additionally, the team utilized specialized probing datasets to measure the model's grasp of code length (LEN), cyclomatic complexity (CPX), and semantic validity (TYP).

Key Insights

The Negative Transfer Trap

The study found that for high-capacity models like UniXcoder, learning one language (e.g., Java) can actively interfere with the performance of another (e.g., Python). Adapter tuning mitigates this "negative transfer" by providing a dedicated space for task-specific learning that does not corrupt the shared weights.

Structural Preservation

Probing experiments revealed that full-model fine-tuning often "washes away" low-level structural information in the deeper layers of the network. Adapters, by contrast, help the model retain its understanding of code complexity and structure, leading to more robust representations.

Efficiency in Low-Resource Scenarios

One of the most striking findings was the efficiency of adapters when data is scarce. Training with as few as 200 samples per language yielded results nearly as competitive as training on the full dataset of hundreds of thousands of samples. This suggests that adapters are highly effective at leveraging the pre-trained model's existing knowledge with minimal supervision.

Results

The adapter-tuned models, mAdapter-CodeT5 and mAdapter-UniXcoder, consistently outperformed both monolingual and standard multilingual baselines:

  • Summarization: mAdapter-CodeT5 achieved an overall BLEU score of 19.74, surpassing the standard multilingual CodeT5 score of 19.41.
  • Search: mAdapter-UniXcoder reached a Mean Reciprocal Rank (MRR) of 75.3, compared to 74.6 for the full-tuned version.
  • Statistical Significance: The improvements were statistically significant (p < 0.001) across nearly all tested languages.

Conclusion and Future Directions

Adapter tuning provides a path forward for scalable, multilingual AI tools in software engineering. By updating less than 1% of parameters, researchers can achieve superior performance while avoiding the pitfalls of catastrophic forgetting.

Future research may explore increasing adapter capacity or integrating explicit code knowledge, such as Abstract Syntax Trees (ASTs), directly into the adapter modules. While initial attempts at using Mixture-of-Experts (MoE) adapters were limited by data constraints for the routing logic, the authors believe this remains a promising area for further optimization in larger-scale environments.

Summary updated

BibTeX
@inproceedings{wang2023one,
  author = {Deze Wang and Boxing Chen and Shanshan Li and Wei Luo and Shaoliang Peng and Wei Dong and Xiangke Liao},
  title = {{One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization}},
  booktitle = {ICSE},
  year = {2023},
  doi = {10.1109/icse48619.2023.00013},
}
Figure