FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion

Qi Guo, Xiaohong Li, Xiaofei Xie, Shangqing Liu, Ze Tang, Ruitao Feng, Junjie Wang, Jidong Ge, Lei Bu

TL;DR

This paper introduces FT2Ra, a novel retrieval-augmented approach for code completion that mimics the fine-tuning process without requiring actual fine-tuning. By theoretically analyzing the impact of fine-tuning on model logits, FT2Ra leverages this insight to improve code completion accuracy, achieving competitive results with state-of-the-art methods and even outperforming fine-tuned models in some scenarios.

What Problem Does This Paper Address?

Large pre-trained models (CPMs) have revolutionized code completion, but fine-tuning them for specific tasks is computationally expensive and data-intensive. Retrieval-augmented language models (RaLMs) offer an alternative by leveraging external knowledge without fine-tuning. However, existing RaLMs often rely on heuristics for selecting and integrating retrieved information, lacking a theoretical grounding. This paper addresses the challenge of effectively using retrieved knowledge to augment code completion, aiming to bridge the gap between heuristic-based RaLMs and the performance benefits of fine-tuning.

Methods

The core of FT2Ra lies in its theoretical analysis of the fine-tuning process. The authors derive that the change in model logits during fine-tuning (Alogits) is a crucial indicator for improving predictions. They propose approximating this Alogits using retrieved nearest neighbors. FT2Ra then integrates this approximated Alogits into the model's original logits to recalibrate its predictions. A key innovation is FT2Ra's iterative retrieval process, which mimics the multi-epoch nature of fine-tuning, allowing for progressive refinement of predictions. The datastore construction involves storing not just ground truth tokens but also their corresponding logits, enabling more sophisticated retrieval in subsequent iterations.

Datasets

The paper evaluates FT2Ra on two widely used benchmarks:

  • kNM-LM dataset: Contains 20 Java projects (10 large-scale, 10 small-scale).
  • CodeXGLUE benchmarks: Includes code samples in Java and Python (JavaCorpus and PY150).

Key Insights and Ideas

  • Alogits as a Fine-Tuning Proxy: The theoretical analysis reveals that the change in logits (Alogits) during fine-tuning is a critical signal for improving predictions. FT2Ra leverages this by approximating Alogits through retrieval, effectively mimicking fine-tuning's effect.
  • Iterative Retrieval for Refinement: FT2Ra's multi-round retrieval process, analogous to fine-tuning epochs, allows for progressive improvement of predictions, leading to better accuracy than single-round retrieval methods.
  • Theoretical Grounding for Retrieval: Unlike many heuristic-based RaLMs, FT2Ra's approach is theoretically motivated, providing a more principled way to select and integrate retrieved information.
  • Competitive Performance Without Fine-Tuning: FT2Ra demonstrates that it can achieve performance comparable to, and sometimes exceeding, models that have undergone actual fine-tuning, offering a computationally efficient alternative.

Key Results

  • Token-Level Completion: FT2Ra achieved a significant improvement of 4.29% accuracy on UniXcoder compared to the best baseline (kNM-LM), reaching 74.22% accuracy.
  • Line-Level Completion: FT2Ra demonstrated a substantial ~2x+ increase in Exact Match (EM) performance on UniXcoder, achieving 26.32% EM compared to the baseline's 13.93%.
  • Outperforming Fine-Tuned Models: In line-level completion, FT2Ra achieved performance comparable to or better than fine-tuned models after several epochs, even when FT2Ra itself was not fine-tuned.
  • Weighting Strategy: The paper found that a distance-based weighting strategy (Rec.) generally outperformed other strategies (Smax, Smax-T, Uni.) when integrated into FT2Ra.
  • Neighbor Sensitivity: FT2Ra showed limited sensitivity to the number of neighbors, suggesting robustness across different settings.
  • Iteration Impact: Multiple retrieval iterations consistently improved FT2Ra's performance, with stabilization typically occurring after around 4 cycles.

Limitations and Future Work

  • Theoretical Framework for Weighting: The authors acknowledge that a concrete theoretical framework for determining the optimal weighting strategy is still lacking, relying on empirical evaluation.
  • Approximation Precision: The approximations inherent in retrieval-augmented methods might affect precision, especially when applied to new models or datasets.
  • Computational Cost: While avoiding fine-tuning, FT2Ra's iterative retrieval process can still be computationally intensive compared to simpler retrieval methods, representing a trade-off between effectiveness and efficiency.
  • Future Work: The authors plan to evaluate FT2Ra on larger models like CodeLLama and CodeGen and further investigate theoretical aspects of weighting strategies. They also suggest that early stopping techniques can be employed to balance computational efficiency and convergence.

Summary updated

Abstract

The rise of code pre-trained models has significantly enhanced various coding tasks, such as code completion, and tools like GitHub Copilot. However, the substantial size of these models, especially large models, poses a significant challenge when it comes to fine-tuning them for specific downstream tasks. As an alternative approach, retrieval-based methods have emerged as a promising solution, augmenting model predictions without the need for fine-tuning. Despite their potential, a significant challenge is that the designs of these methods often rely on heuristics, leaving critical questions about what information should be stored or retrieved and how to interpolate such information for augmenting predictions. To tackle this challenge, we first perform a theoretical analysis of the fine-tuning process, highlighting the importance of delta logits as a catalyst for improving model predictions. Building on this insight, we develop a novel retrieval-based method, FT2Ra, which aims to mimic genuine fine-tuning. While FT2Ra adopts a retrieval-based mechanism, it uniquely adopts a paradigm with a learning rate and multi-epoch retrievals, which is similar to fine-tuning.In token-level completion, which represents a relatively easier task, FT2Ra achieves a 4.29% improvement in accuracy compared to the best baseline method on UniXcoder. In the more challenging line-level completion task, we observe a substantial more than twice increase in Exact Match (EM) performance, indicating the significant advantages of our theoretical analysis. Notably, even when operating without actual fine-tuning, FT2Ra exhibits competitive performance compared to the models with real fine-tuning.
BibTeX
@inproceedings{guo2024ft,
  author = {Qi Guo and Xiaohong Li and Xiaofei Xie and Shangqing Liu and Ze Tang and Ruitao Feng and Junjie Wang and Jidong Ge and Lei Bu},
  title = {{FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion}},
  booktitle = {ISSTA},
  year = {2024},
  doi = {10.1145/3650212.3652130},
}
Figure