TL;DR
This paper surveys the rapidly advancing field of pre-trained models for source code (CodePTMs) and their applications in software engineering (SE). It highlights how these models, inspired by natural language processing (NLP) advancements, are achieving state-of-the-art results on various SE tasks, bridging the gap between deep learning and software development.
What Problem Does This Paper Address?
Traditionally, applying deep learning to software engineering (SE) tasks required large, often expensive, annotated datasets for training models from scratch. This "data annotation bottleneck" limited the widespread adoption of deep learning in SE. Inspired by the success of pre-trained language models (PLMs) in NLP, this paper addresses the challenge of effectively leveraging deep learning for SE by exploring the development and application of pre-trained models specifically designed for source code (CodePTMs). The authors aim to raise awareness among AI researchers about the significant impact of these models on SE, a crucial application domain for AI.
Methods
The paper provides a comprehensive survey of CodePTMs, categorizing them along four key dimensions:
- Architecture: CodePTMs are built upon existing deep learning architectures, primarily variations of Recurrent Neural Networks (RNNs) like Long Short-Term Memory (LSTM) and Transformer-based models (Transformer, Transformer-Encoder, Transformer-Decoder).
- Modality: Models are classified as unimodal (using only code), bimodal (code + natural language, or code + structure), or multimodal (code + natural language + structure). The integration of natural language (e.g., comments, documentation) and structural information (e.g., Abstract Syntax Trees - ASTs, Control Flow Graphs - CFGs) is crucial for improving performance.
- Pre-Training Tasks: These tasks are divided into two categories:
- NLP-originated: Language Modeling (LM), Masked Language Modeling (MLM), Denoising Auto-Encoding (DAE), and Contrastive Learning (CTL).
- SE-specific: Code-Aware (CA), Structure-Aware (SA), and Cross-Modal-Aware (CMA) tasks, which are further broken down into Code-NL (CN), Code-Structure (CS), and Code-NL-Structure (CNS).
- Programming Languages (PL): Models are either monolingual (trained on a single PL) or multilingual (trained on multiple PLs).
The paper also discusses different strategies for combining pre-training tasks, including simultaneous, sequential, and alternate learning.
Datasets
The paper references numerous datasets used for training and evaluating CodePTMs. These include:
- General Code Repositories: GitHub Repos (e.g., JS GitHub Repos), BigQuery (large-scale code collections), CodeSearchNet (functions paired with docstrings).
- Cross-Language Datasets: CLCDSA (functionally similar code snippets across multiple PLs).
- Task-Specific Datasets: Various datasets are mentioned for specific SE tasks like bug detection, code completion, and summarization, often identified by IDs in Table 1.
The paper does not introduce new benchmark datasets but rather surveys existing ones and their usage.
Key Insights and Ideas
- Pre-training is Key: The core idea is that pre-training on large amounts of code (often using self-supervised tasks) allows models to learn general code representations, which can then be fine-tuned for specific SE tasks with much less labeled data.
- Code is Different from Natural Language: While NLP techniques are foundational, source code has unique characteristics (e.g., structured syntax, mixed code and natural language) that necessitate specialized pre-training tasks and architectures for optimal performance.
- Multimodality Matters: Incorporating natural language (comments, documentation) and structural information (ASTs, CFGs) significantly boosts CodePTM performance on many SE tasks. However, the benefit of NL depends on its presence in the downstream task.
- Architecture Choice is Task-Dependent: Transformer-Encoder (TE) based models tend to excel at understanding tasks, while Transformer-Decoder (TD) and full Transformer (TF) models are better suited for generation tasks.
- Pre-training Task Similarity: Aligning pre-training tasks with downstream tasks (e.g., using understanding-focused tasks for understanding tasks) leads to better transfer learning.
Key Results
The paper demonstrates that CodePTMs consistently achieve state-of-the-art (SOTA) results across a wide range of SE tasks, including bug detection, code completion, code summarization, and more. Table 5 shows that CodePTMs achieve significant relative error reduction rates, ranging from 0.9% to 78.7%, compared to non-pre-trained approaches. This suggests that CodePTMs are a highly effective paradigm for advancing SE capabilities.
Limitations and Future Work
- Limitations:
- Current CodePTMs largely rely on NLP-based tokenization and embedding methods, which may not fully capture code-specific nuances.
- Many pre-training tasks are still adaptations of NLP tasks, lacking a fundamental overhaul for code.
- The evaluation of CodePTMs is often limited to specific tasks and datasets, hindering a comprehensive understanding of their strengths and weaknesses.
- Current models primarily learn code "form" rather than "functionality."
- Future Work:
- Develop code-specific tokenization and embedding methods.
- Design novel pre-training tasks that better exploit code structure, branches, and identifier usage.
- Explore learning code functionality beyond just its form.
- Investigate more efficient adaptation methods beyond full fine-tuning (e.g., model reprogramming, prompt tuning, compression).
- Develop specialized CodePTMs for different classes of SE tasks (e.g., understanding vs. generation).
- Conduct unified evaluations and qualitative analyses to better understand model behaviors and common errors.
Summary updated