A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniques

Xiaotao Song, Hailong Sun, Xu Wang, Jiafei Yan

Abstract

As an integral part of source code files, code comments help improve program readability and comprehension. However, developers sometimes do not comment on their program code adequately due to the incurred extra efforts, lack of relevant knowledge, or unawareness of the importance of code commenting. As a result, code comments can be inadequate, absent, or even mismatched with source code, which affects the understanding, reuse, and maintenance of software. To solve these problems, researchers have focused on generating code comments automatically. This work conducts a survey of automatic code commenting research, analyzing challenges, research frameworks, and the classification of representative algorithms, including their design principles, strengths, and weaknesses. We also provide an overview of quality assessment and summarize future directions for the field.

Introduction

Software maintenance and reuse are heavily dependent on high-quality documentation. Yet, developers often skip commenting because it is time-consuming or they lack context during active development. The challenge of Automatic Code Commenting is not merely describing what a line of code does, but explaining the developer's intent. Bridging the semantic gap between highly structured programming languages and fluid natural language requires an understanding of both program flow and logic.

Taxonomy of Generation Algorithms

Research into automatic comment generation is categorized into three primary algorithmic families: Information Retrieval (IR), Deep Learning (DL), and Heuristic/Structural models.

Taxonomy of automatic comment generation algorithms, categorizing IR-based, Deep NN-based, and other specialized techniques.
  1. Information Retrieval (IR) Based: These methods act as search engines, using techniques like the Vector Space Model (VSM) or Latent Semantic Indexing (LSI) to find similar code snippets in large repositories (like GitHub) and "borrowing" their existing comments.
  2. Deep Learning (DL) Based: This represents the current state-of-the-art, treating commenting as a Machine Translation problem (e.g., translating Python to English). Key architectures include Encoder-Decoder (Seq2Seq) models, Attention Mechanisms to focus on specific code elements, and Tree-based models (RvNN/CNN) that process Abstract Syntax Trees (ASTs) to capture logic hierarchies.
  3. Heuristic & Structural Models: These use specialized models like the Software Word Usage Model (SWUM) to extract "action-theme" relationships from code identifiers to build template-based descriptions.

Data Sources and Evaluation

There is currently no single "ImageNet" for code comments, making cross-paper comparisons difficult. Researchers typically draw from open-source repositories (GitHub), Q&A sites (Stack Overflow), and developer communications (bug reports).

Evaluation remains a significant hurdle. While translation metrics like BLEU and METEOR are standard, they often fail to capture the true semantic utility of a comment. A comment might have a low BLEU score but still be perfectly accurate and helpful to a human. Consequently, human evaluation for informativeness and interpretability remains the gold standard.

Evolution of the Field

The landscape of code comment generation has shifted dramatically over the last decade. Early research was predominantly IR-based or relied on heuristic templates. However, as shown in the publication trends, there has been an explosion in Deep Neural Network (DNN) approaches since 2015.

Publication trends from 2010 to 2018 showing the rapid growth of Deep Neural Network-based algorithms in code comment generation.

Hybrid models—those that combine structural analysis (like ASTs) with sequence-to-sequence learning—consistently outperform models that treat code as plain text. While DL-based methods are better at generalizing, they can occasionally produce "hallucinations"—grammatically correct English that is factually incorrect regarding the code's logic.

Future Directions

To advance the field, researchers must address dataset fragmentation by establishing unified, high-quality benchmarks. Furthermore, current models often suffer from "context blindness," looking only at single functions. Future work needs to incorporate "cross-file" context to understand how a specific function relates to the broader system. The next major leap is expected to come from better "fusion" models that combine the reliability of IR-based retrieval with the creative generation capabilities of Deep Learning.

Summary updated

Abstract

As an integral part of source code files, code comments help improve program readability and comprehension. However, developers sometimes do not comment on their program code adequately due to the incurred extra efforts, lack of relevant knowledge, unawareness of the importance of code commenting or some other factors. As a result, code comments can be inadequate, absent or even mismatched with source code, which affects the understanding, reusing and the maintenance of software. To solve these problems of code comments, researchers have been concerned with generating code comments automatically. In this work, we aim at conducting a survey of automatic code commenting researches. First, we generally analyze the challenges and research framework of automatic generation of program comments. Second, we present the classification of representative algorithms, the design principles, strengths and weaknesses of each category of algorithms. Meanwhile, we also provide an overview of the quality assessment of the generated comments. Finally, we summarize some future directions for advancing the techniques of automatic generation of code comments and the quality assessment of comments.
BibTeX
@article{song2019survey,
  author = {Xiaotao Song and Hailong Sun and Xu Wang and Jiafei Yan},
  title = {{A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniques}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2931579},
}
Figure