A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniques

Xiaotao Song, Hailong Sun, Xu Wang, Jiafei Yan

TL;DR

This paper provides a comprehensive map of the "code-to-comment" landscape, tracing the evolution from simple search-and-retrieve methods to sophisticated deep learning architectures. It highlights how researchers are treating source code as a language to be translated, while grappling with the unique challenge of capturing both the logic and the intent of a developer.

What Problem Does This Paper Address?

We’ve all been there: you inherit a legacy codebase with zero comments, or worse, comments that contradict the code. While documentation is vital for software maintenance and reuse, developers often skip it because it’s time-consuming or they lack the "big picture" context during active development.

The authors address the challenge of Automatic Code Commenting. This isn't just about describing what a line does (e.g., "increments x"); it’s about explaining the intent (e.g., "updates the retry counter"). The fundamental hurdle is that programming languages are highly structured and nested, while natural language is unstructured and fluid. Bridging this semantic gap requires more than just a dictionary; it requires an understanding of program flow and developer logic.

Methods

The paper categorizes the research into three main algorithmic "families":

  1. Information Retrieval (IR) Based: These are the "search engines" of commenting. They use techniques like VSM (Vector Space Model) or LSI (Latent Semantic Indexing) to find similar code snippets in a large database (like GitHub) and "borrow" their comments. If your code looks like a standard "file-open" routine, the system finds a similar routine and suggests its existing comment.
  2. Deep Learning (DL) Based: This is the current state-of-the-art. Researchers treat code commenting as a Machine Translation problem (translating Python to English). Key architectures include:
  3. Encoder-Decoder (Seq2Seq): An encoder compresses code into a vector, and a decoder "unpacks" it into a sentence.
  4. Attention Mechanisms: These allow the model to "focus" on specific parts of the code (like a variable name or a specific loop) when generating a specific word in the comment.
  5. Tree-based Models (RvNN/CNN): Unlike standard text, code has a tree structure (AST). Some models specifically process these trees to capture the hierarchy of the logic.
  6. Heuristic & Structural Models: These use specialized models like SWUM (Software Word Usage Model) to extract the "action-theme" (verb-noun) relationships from code identifiers to build a template-based description.

Datasets

The survey notes that there is no single "ImageNet" for code comments, which makes comparing different papers difficult. However, most researchers draw from three sources:

  • Open Source Repositories: Massive crawls of GitHub (specifically Java and Python projects).
  • Q&A Sites: Mining Stack Overflow to find "code-description" mappings.
  • Developer Communications: Using bug reports and mailing lists to understand the context behind specific code changes.

Key Insights and Ideas

  • The Structural Advantage: Code is not just a sequence of words; it’s a graph. The most successful recent models are those that don't just look at code as "text" but incorporate Abstract Syntax Trees (ASTs) to understand the nesting and dependencies of the logic.
  • The "Intent" Gap: IR-based methods are great at describing common patterns but fail on unique, proprietary logic. DL-based methods are better at generalizing but can sometimes produce "hallucinations"—grammatically correct English that is factually wrong about the code.
  • Evaluation is Hard: We currently use translation metrics like BLEU and METEOR to grade comments. However, a comment can have a low BLEU score (because it uses different words) but still be perfectly accurate and helpful to a human.

Key Results

The survey highlights a massive shift in the field:

  • The DL Explosion: Before 2014, almost all research was IR-based. By 2018, nearly all top-tier papers focused on Deep Neural Networks.
  • Performance Gains: Hybrid models—those that combine structural analysis (like ASTs) with sequence-to-sequence learning—consistently outperform models that treat code as plain text.
  • Human vs. Machine: While automatic metrics are improving, human evaluation remains the "gold standard," measuring nuances like Informativeness (does it tell me something I don't already know?) and Interpretability.

Limitations and Future Work

  • Dataset Fragmentation: The lack of a unified, high-quality benchmark dataset is the biggest bottleneck. Different papers use different "cleanliness" standards for their GitHub crawls, making head-to-head comparisons nearly impossible.
  • Context Blindness: Most current models only look at a single function or block. Future work needs to look at "cross-file" context—how does this function relate to the rest of the system?
  • Synergy: The authors suggest that the next big leap will come from better "fusion" models that combine the reliability of IR-based retrieval with the creative generation of Deep Learning.

Summary updated

Abstract

As an integral part of source code files, code comments help improve program readability and comprehension. However, developers sometimes do not comment on their program code adequately due to the incurred extra efforts, lack of relevant knowledge, unawareness of the importance of code commenting or some other factors. As a result, code comments can be inadequate, absent or even mismatched with source code, which affects the understanding, reusing and the maintenance of software. To solve these problems of code comments, researchers have been concerned with generating code comments automatically. In this work, we aim at conducting a survey of automatic code commenting researches. First, we generally analyze the challenges and research framework of automatic generation of program comments. Second, we present the classification of representative algorithms, the design principles, strengths and weaknesses of each category of algorithms. Meanwhile, we also provide an overview of the quality assessment of the generated comments. Finally, we summarize some future directions for advancing the techniques of automatic generation of code comments and the quality assessment of comments.
BibTeX
@article{song2019survey,
  author = {Xiaotao Song and Hailong Sun and Xu Wang and Jiafei Yan},
  title = {{A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniques}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2931579},
}

Loading related content...

Figure