A deep language model for software code

Hoa Khanh Dam, Truyen Tran, Trang Pham

Abstract: Existing language models such as n-grams for software code often fail to capture a long context where dependent code elements scatter far apart. In this paper, we propose a novel approach to build a language model for software code to address this particular issue. Our language model, partly inspired by human memory, is built upon the powerful deep learning-based Long Short Term Memory architecture that is capable of learning long-term dependencies which occur frequently in software code. Results from our intrinsic evaluation on a corpus of Java projects have demonstrated the effectiveness of our language model. This work contributes to realizing our vision for DeepSoft, an end-to-end, generic deep learning-based framework for modeling software and its development process.

Introduction

Traditional language models for code, such as n-grams, rely on a very short "memory," typically predicting the next token based only on the previous 2 to 5 tokens. However, software code is highly structured and full of long-term dependencies. For example, a try block might require a catch block dozens of lines later, or a file opened at the start of a function must be closed at the end.

Standard Recurrent Neural Networks (RNNs) were an improvement but suffered from the "vanishing gradient" problem, making it difficult for them to actually learn these long-distance connections. This paper seeks to bridge that gap by applying Long Short-Term Memory (LSTM) networks to capture the "semantics hidden deeply in source code."

Methodology

The authors treat source code as a sequence of tokens and apply a specific deep learning architecture to model the temporal dependencies within the code.

LSTM sequence modeling for Java code tokens
  • LSTM Architecture: Unlike standard RNNs, LSTMs use a "memory cell" and a series of gates (input, forget, and output) to regulate the flow of information. This allows the model to "forget" irrelevant local variables and "remember" important structural elements (like an open stream) over hundreds of tokens.
  • Tokenization & Embedding: Code is lexically analyzed into tokens. Each token is then mapped into a continuous vector space (an embedding), allowing the model to learn mathematical relationships between similar code elements.
  • Noise Contrastive Estimation (NCE): To handle the massive vocabularies found in large software repositories, the authors used NCE. This technique avoids the massive computational cost of a standard "softmax" layer by turning the prediction task into a binary classification problem.
  • Regularization: They employed Dropout and Early Stopping to prevent the model from simply memorizing the training data (overfitting).

Experimental Setup

The researchers curated a dataset from 10 popular open-source Java projects cloned from GitHub, including Ant, Cassandra, Lucene, and Maven. The final corpus consisted of over 6.1 million code tokens with a vocabulary of 81,213 unique tokens. During preprocessing, they replaced specific integers, strings, and rare tokens with generic tags (e.g., <num>, <str>, <unk>) to keep the vocabulary manageable.

Key Results

The model was evaluated using perplexity, a standard metric where a lower score indicates a better, more "certain" model. The results demonstrated that the LSTM model outperformed simple RNNs in every test case.

Comparison of RNN and LSTM performance across varying sequence lengths and embedding dimensions
  • Impact of Context: The performance gap between LSTMs and simple RNNs widens as the context window increases. For short sequences (10 tokens), the LSTM was 4.7% better than a simple RNN. For long sequences (500 tokens), the improvement jumped to 27.2%.
  • Scaling Benefits: When the model size (embedding dimensions) was increased for a fixed sentence length of 100, the LSTM showed a massive 37.9% improvement over the baseline RNN. This suggests that LSTMs scale much more effectively as the complexity of the model increases.

Discussion and Future Work

The most significant insight is the "Long-Range" advantage: LSTMs are fundamentally better at "reading" long files where dependencies are scattered. This paper is presented as a building block for a larger framework called "DeepSoft," intended to model the entire software development process using deep learning.

However, several challenges remain for future research:

  • Extrinsic Evaluation: Future work needs to test the model on tasks like actual code suggestion, bug localization, or vulnerability detection rather than just perplexity.
  • Structural Awareness: While LSTMs handle sequences well, they do not explicitly account for the tree-like structure of code (Abstract Syntax Trees).
  • Vocabulary Constraints: Handling the "out-of-vocabulary" problem (new variable names the model hasn't seen before) remains a challenge in software modeling.

Summary updated

Abstract

Existing language models such as n-grams for software code often fail to capture a long context where dependent code elements scatter far apart. In this paper, we propose a novel approach to build a language model for software code to address this particular issue. Our language model, partly inspired by human memory, is built upon the powerful deep learning-based Long Short Term Memory architecture that is capable of learning long-term dependencies which occur frequently in software code. Results from our intrinsic evaluation on a corpus of Java projects have demonstrated the effectiveness of our language model. This work contributes to realizing our vision for DeepSoft, an end-to-end, generic deep learning-based framework for modeling software and its development process.
BibTeX
@inproceedings{dam2016deep,
  author = {Hoa Khanh Dam and Truyen Tran and Trang Pham},
  title = {{A deep language model for software code}},
  booktitle = {Deakin},
  year = {2016},
}
Figure