Code Completion with Neural Attention and Pointer Networks

Jian Li, Yue Wang, Michael R. Lyu, Irwin King

Abstract

Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.

Introduction

Traditional code completion tools often rely on static type information, which is frequently absent in dynamically-typed languages like Python and JavaScript. While neural language models, specifically Long Short-Term Memory (LSTM) networks, offer a promising alternative by learning from large-scale codebases, they face two primary challenges: the "hidden state bottleneck" in capturing long-range dependencies and the "Out-of-Vocabulary" (OoV) problem. Because neural models typically use a fixed vocabulary for efficiency, rare or unique variable names are often replaced with a generic UNK token, rendering the completion tool ineffective for many practical scenarios.

This paper addresses these issues by leveraging the observation that while variable names may be globally rare, they are frequently repeated within a local context (e.g., the same file or function). We introduce a Pointer Mixture Network that combines a global RNN with a local pointer mechanism to effectively handle both common keywords and unique identifiers.

Methodology

Program Representation

To capture the structural information inherent in source code, the model represents programs as Abstract Syntax Trees (ASTs). These trees are flattened into a sequence of nodes, where each node contains both a type (e.g., Assign, For) and a value (e.g., a variable name or constant). This transformation allows the model to process code as a sequence while maintaining structural cues.

The process of parsing code into an AST and flattening it into a sequence of nodes.

Structural Attention

When an AST is flattened, hierarchical relationships can be obscured. To mitigate this, we introduce a "parent attention" mechanism. This allows the model to explicitly attend to the hidden state of a node's parent in the original AST, providing critical context about the current code block (e.g., knowing the model is currently within the body of a For loop).

Pointer Mixture Network

The core innovation is the Pointer Mixture Network, which consists of two main components:

  1. Global RNN Component: Predicts the next token from a fixed, pre-defined vocabulary of common keywords and identifiers.
  2. Local Pointer Component: Uses attention scores to identify and copy tokens from the recent local context.

A sigmoid-based gating mechanism, or "switcher," determines whether the next token should be generated from the global vocabulary or copied from the local context via the pointer.

Architecture of the Pointer Mixture Network showing the RNN and Pointer components.

Experimental Evaluation

The proposed model was evaluated on two large-scale datasets from GitHub, consisting of 150,000 JavaScript files and 150,000 Python files. The experiments focused on predicting both the type and the value of the next AST node.

  • Accuracy Improvements: On the JavaScript dataset, the Pointer Mixture Network achieved 81.0% accuracy for value prediction, significantly outperforming vanilla LSTMs (78.6%).
  • Type Prediction: The attentional model reached 88.6% accuracy for predicting node types, setting a new benchmark for structural code completion.
  • Vocabulary Efficiency: The pointer mechanism proved particularly robust when using smaller global vocabularies, as it could effectively "copy" missing words from the local context that standard models would otherwise label as UNK.

Conclusion

By combining structural attention with a pointer mixture mechanism, this research significantly improves the accuracy and utility of code completion for dynamically-typed languages. The ability to copy identifiers from local context effectively solves the out-of-vocabulary problem, while parent attention preserves the hierarchical nature of source code within a sequential model.

Summary updated

Abstract

Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.
BibTeX
@inproceedings{li2018code,
  author = {Jian Li and Yue Wang and Michael R. Lyu and Irwin King},
  title = {{Code Completion with Neural Attention and Pointer Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=HkEqaNMubH},
}
Figure