Code Completion with Neural Attention and Pointer Networks

Jian Li, Yue Wang, Irwin King, Michael R. Lyu

Abstract: Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.

1. Introduction

Modern Integrated Development Environments (IDEs) rely heavily on code completion to enhance developer productivity. While statically-typed languages like Java benefit from compile-time type information, dynamically-typed languages such as JavaScript and Python present a greater challenge. Recent research has treated source code as a sequence of tokens, applying Recurrent Neural Networks (RNNs) for prediction. However, these models face two primary hurdles: the "hidden state bottleneck" for long-range dependencies and the "unknown word problem," where rare identifiers are replaced by a generic UNK token. This paper proposes a novel architecture that leverages the hierarchical structure of code and local repetition to overcome these limitations.

2. Methodology

The proposed approach represents source code as flattened Abstract Syntax Trees (ASTs). Each node in the sequence contains both a Type (e.g., IfStatement) and a Value (e.g., my_variable).

2.1 AST Parsing and Flattening

To prepare the data for sequential modeling, the source code is parsed into an AST and then flattened. This process, illustrated in Figure 1, transforms the hierarchical tree into a sequence of nodes while maintaining syntactic information.

The process of parsing source code into an AST and flattening it into a sequence of nodes.

2.2 Parent-Aware Attention

Standard attention mechanisms often lose hierarchical context during AST flattening. To mitigate this, the authors introduce a parent-aware attention mechanism. This tailored approach explicitly retrieves the hidden state of the current node's parent in the original AST structure, providing critical syntactic cues for predicting the next node.

2.3 Pointer Mixture Network

To address the out-of-vocabulary (OoV) problem, the authors exploit the tendency of developers to repeat variables locally. The Pointer Mixture Network (Figure 2) features a dual-headed design:

  • Global RNN Component: Predicts common tokens from a fixed, global vocabulary.
  • Local Pointer Component: "Points" to specific locations in the recent context window to copy tokens directly.
  • Learned Switcher: A sigmoid-based gate that dynamically weighs the outputs of these two components based on the current context.
Architecture of the Pointer Mixture Network featuring the RNN component, Pointer component, and the dynamic switcher $s_t$.

3. Experimental Evaluation

The models were trained and evaluated on two large-scale GitHub datasets: JavaScript (10.7M queries) and Python (6.2M queries). To evaluate robustness against OoV tokens, the global vocabulary size was varied (1k, 10k, and 50k).

4. Results and Discussion

The experimental results demonstrate significant improvements over baseline models:

  • AST Node Type Prediction: The Attentional LSTM achieved state-of-the-art accuracy of 88.6% on JavaScript and 80.6% on Python.
  • AST Node Value Prediction: The Pointer Mixture Network reached 81.0% accuracy on JavaScript and 70.1% on Python (with a 50k vocabulary).
  • OoV Performance: The most dramatic gains were observed in restricted vocabulary settings. For the JS 1k dataset, accuracy improved from 69.9% (Vanilla LSTM) to 73.2%, highlighting the pointer network's ability to recover rare identifiers from local context.

5. Conclusion

This paper introduces a robust framework for code completion in dynamically-typed languages by integrating AST-based representation, parent-aware attention, and a pointer mixture network. By effectively capturing both global syntactic patterns and local identifier repetitions, the model significantly outperforms standard neural language models, particularly in handling out-of-vocabulary tokens.

Summary updated

Abstract

Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.
BibTeX
@inproceedings{li2018code,
  author = {Jian Li and Yue Wang and Irwin King and Michael R. Lyu},
  title = {{Code Completion with Neural Attention and Pointer Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=HkEqaNMubH},
}
Figure