FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion

Qi Guo, Xiaohong Li, Xiaofei Xie, Shangqing Liu, Ze Tang, Ruitao Feng, Junjie Wang, Jidong Ge, Lei Bu

TL;DR

This paper introduces FT2Ra, a novel retrieval-augmented approach for code completion that mimics the fine-tuning process without requiring actual fine-tuning. By theoretically analyzing how fine-tuning improves model predictions, FT2Ra leverages "delta logits" (changes in model outputs) to enhance code completion accuracy, achieving significant improvements over existing methods and even rivaling fine-tuned models.

What Problem Does This Paper Address?

Large pre-trained models (CPMs) have revolutionized code completion, but fine-tuning them for specific tasks is computationally expensive and data-intensive. Retrieval-augmented language models (RaLMs) offer an alternative by leveraging external knowledge without fine-tuning. However, existing RaLMs often rely on heuristics for selecting and integrating retrieved information, lacking a theoretical foundation. This paper aims to bridge this gap by providing a theoretical understanding of fine-tuning's effectiveness and using it to design a more principled retrieval-augmented method for code completion.

Methods

The core of FT2Ra lies in its theoretical analysis of the fine-tuning process. The authors derive that the change in model logits (the raw output before the final probability calculation) during fine-tuning, termed "delta logits" (Alogits), is crucial for improving predictions. FT2Ra approximates these delta logits by retrieving relevant code snippets and using their information to adjust the original model's logits.

Key methodological contributions include:

  • Theoretical Analysis of Fine-tuning: The paper mathematically models how fine-tuning impacts model parameters and, consequently, the logits. This analysis highlights the importance of Alogits.
  • Alogits Approximation: FT2Ra approximates Alogits by considering the difference between the ground truth and predicted probabilities of retrieved neighbors.
  • Iterative Retrieval: Unlike many existing methods, FT2Ra employs an iterative retrieval process, mimicking the multi-epoch nature of fine-tuning. This allows for progressive refinement of predictions.
  • Weighting Strategy: The method incorporates a distance-based weighting strategy for retrieved neighbors, giving more importance to closer matches.

Datasets

The study utilizes two primary datasets for evaluation:

  • kNM-LM dataset: This dataset comprises 20 Java projects, split into large-scale and small-scale subsets.
  • CodeXGLUE benchmarks: This includes code samples in both Java and Python.

The authors followed existing settings for data preparation and splitting, using the training data for fine-tuning experiments and the test data for evaluating FT2Ra and baselines.

Key Insights and Ideas

  • The Power of Delta Logits: The theoretical analysis reveals that the change in logits during fine-tuning is a key driver of performance improvement, providing a more direct signal than just the final probabilities.
  • Mimicking Fine-tuning: FT2Ra demonstrates that by approximating these critical "delta logits" and employing an iterative retrieval process, one can achieve performance comparable to actual fine-tuning without the associated computational costs.
  • Iterative Refinement is Key: The multi-round retrieval strategy, analogous to fine-tuning epochs, significantly boosts performance, showing that repeated refinement of predictions is beneficial.
  • Theoretical Grounding for Retrieval: FT2Ra moves beyond heuristic-based retrieval by providing a theoretically motivated approach, making the selection and integration of retrieved information more principled.

Key Results

FT2Ra shows strong performance across various scenarios:

  • Token-Level Completion: On UniXcoder, FT2Ra achieved a 4.29% improvement in accuracy compared to the best baseline (kNM-LM), reaching 74.22% accuracy.
  • Line-Level Completion: FT2Ra demonstrated a substantial improvement, achieving more than a 2x increase in Exact Match (EM) performance on UniXcoder, reaching 26.32% compared to the baseline's 1.63%.
  • Comparison to Fine-tuning: FT2Ra, without any fine-tuning, achieved performance comparable to models fine-tuned for several epochs. For instance, it matched fine-tuned CodeGPT after about 4 epochs and fine-tuned UniXcoder after about 7 epochs.
  • Efficiency: While FT2Ra is slightly slower than some input-retrieval methods due to its output-retrieval and iterative nature, it offers a significant performance gain for a modest increase in prediction time.

Limitations and Future Work

  • Weighting Strategy: The paper acknowledges that while their distance-based weighting strategy is effective, a concrete theoretical framework for determining optimal weights is still an open area. They empirically evaluated four strategies and selected the most effective one.
  • Approximation Accuracy: The inherent approximations in retrieval-augmented methods might impact precision, especially when applied to new models or datasets.
  • Large Models: The study focused on CodeGPT and UniXcoder due to computational constraints. Evaluating FT2Ra on even larger models like CodeLLama and CodeGen is suggested for future work.
  • Theoretical Framework for Weighting: The authors plan to further investigate the theoretical underpinnings of weighting strategies.
  • Transparency: The authors have made their codebase, datasets, and experimental results publicly available to enable independent verification.

Summary updated

Abstract

The rise of code pre-trained models has significantly enhanced various coding tasks, such as code completion, and tools like GitHub Copilot. However, the substantial size of these models, especially large models, poses a significant challenge when it comes to fine-tuning them for specific downstream tasks. As an alternative approach, retrieval-based methods have emerged as a promising solution, augmenting model predictions without the need for fine-tuning. Despite their potential, a significant challenge is that the designs of these methods often rely on heuristics, leaving critical questions about what information should be stored or retrieved and how to interpolate such information for augmenting predictions. To tackle this challenge, we first perform a theoretical analysis of the fine-tuning process, highlighting the importance of delta logits as a catalyst for improving model predictions. Building on this insight, we develop a novel retrieval-based method, FT2Ra, which aims to mimic genuine fine-tuning. While FT2Ra adopts a retrieval-based mechanism, it uniquely adopts a paradigm with a learning rate and multi-epoch retrievals, which is similar to fine-tuning.In token-level completion, which represents a relatively easier task, FT2Ra achieves a 4.29% improvement in accuracy compared to the best baseline method on UniXcoder. In the more challenging line-level completion task, we observe a substantial more than twice increase in Exact Match (EM) performance, indicating the significant advantages of our theoretical analysis. Notably, even when operating without actual fine-tuning, FT2Ra exhibits competitive performance compared to the models with real fine-tuning.
BibTeX
@inproceedings{guo2024ft,
  author = {Qi Guo and Xiaohong Li and Xiaofei Xie and Shangqing Liu and Ze Tang and Ruitao Feng and Junjie Wang and Jidong Ge and Lei Bu},
  title = {{FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion}},
  booktitle = {ISSTA},
  year = {2024},
  doi = {10.1145/3650212.3652130},
}

Loading related content...

Figure