TL;DR
This paper introduces CodeSum, a novel approach to automatically generate natural language summaries for source code by treating it as a machine translation problem. CodeSum leverages an attention-based Seq2Seq model that processes Abstract Syntax Trees (ASTs) traversed using a new Structure-based Traversal (SBT) method, significantly outperforming existing methods.
What Problem Does This Paper Address?
Programmers spend a considerable amount of time understanding source code, and reading comments or documentation (like JavaDoc) is a crucial aid. However, these human-written summaries are often missing, incomplete, or outdated. The task of code summarization aims to automatically generate these natural language descriptions to improve code comprehension, code search, and categorization. While previous work has explored this, existing methods often treat code as plain text, ignoring its inherent structure, or produce overly short summaries. This paper addresses the need for a more robust and structurally aware code summarization system that can generate more informative and accurate descriptions.
Methods
CodeSum employs an attention-based Sequence-to-Sequence (Seq2Seq) neural network, a standard architecture for machine translation tasks. The key innovation lies in how it processes the source code:
- Abstract Syntax Trees (ASTs): Instead of feeding raw code text, CodeSum first parses the code into its AST representation. ASTs capture the hierarchical structure of the code, which is crucial for understanding its logic.
- Structure-based Traversal (SBT): To convert the AST into a format suitable for the Seq2Seq model, the authors propose a new traversal method called SBT. SBT uses brackets to represent the tree structure and traverses subtrees recursively, generating a sequence that preserves the AST's structure and avoids ambiguity. This is a departure from simpler traversal methods like Depth-First Search (DFS).
- Attention Mechanism: The Seq2Seq model uses an attention mechanism to allow the decoder to focus on relevant parts of the encoded AST sequence when generating each word of the summary. This helps in aligning code elements with their corresponding descriptive words.
Datasets
The authors conducted experiments on three large-scale corpora:
- Java Corpus: This is a newly proposed industry code corpus extracted from GitHub.
- C# and SQL Corpora: These datasets were used by a previous state-of-the-art method (CODENN) for evaluation.
For training and evaluation, they extracted Java methods and the first sentence of their JavaDoc descriptions. They filtered out non-word descriptions, setter/getter/constructor/test methods, and split the data into training, validation, and testing sets in an 8:1:1 ratio.
Key Insights and Ideas
- Code as Translation: Framing code summarization as a machine translation problem is a powerful paradigm, allowing the application of successful NLP techniques like Seq2Seq and attention.
- Structure Matters: Explicitly encoding the structural information of code via ASTs, rather than treating it as a flat sequence of tokens, significantly improves summarization accuracy.
- SBT's Advantage: The proposed SBT traversal method is more effective than traditional DFS for representing ASTs because it better captures structural semantics and maintains unambiguity, leading to better alignment with natural language summaries.
- Beyond Short Summaries: CodeSum can generate more descriptive summaries compared to prior work that focused on very short, name-like summaries.
Key Results
- Outperforming State-of-the-Art: CodeSum significantly outperforms the previous state-of-the-art method, CODENN, across different programming languages.
- BLEU Scores:
- On the Java corpus, CodeSum with SBT achieved a BLEU-4 score of 38.17%, compared to CODENN's 25.3%.
- Even without ASTs (using plain text input), CodeSum's Seq2Seq model achieved 34.87%, a substantial improvement over CODENN.
- On C# and SQL, CodeSum also showed superior performance, achieving 30.00% and 30.94% respectively, compared to CODENN's 20.4% and 17.0%.
- Impact of SBT: Using SBT for AST traversal improved CodeSum's BLEU score from 36.01% (with DFS) to 38.17%.
- Code and Summary Lengths: CodeSum maintains high accuracy across various code and summary lengths, whereas CODENN's accuracy degrades significantly with longer summaries.
Limitations and Future Work
- User Identifiers: The paper acknowledges that CodeSum's performance is limited when summaries are highly dependent on user-defined identifiers, as these can vary greatly in style even when conveying similar meanings.
- Future Directions: The authors plan to develop better models to handle user identifiers and explore other applications of deep learning in software engineering. They also suggest further investigation into the other applications of deep learning approaches in software engineering.
Summary updated