TL;DR
This paper introduces a tool that automatically extracts and processes code comments and commit messages from open-source repositories, creating a large-scale corpus of these "source code annotations." This work addresses the understudied linguistic domain of how natural language is used to explain code, providing a valuable resource for future research in computational linguistics and software engineering.
What Problem Does This Paper Address?
While many specialized sublanguages of human language have been extensively studied (like cooking recipes), the language used to annotate source code – comments and commit messages – remains largely unexplored from a linguistic perspective. This is a significant gap because these annotations are crucial for understanding, maintaining, and developing software. They represent a unique intersection of natural language and formal programming languages, offering insights into how we communicate about complex technical concepts. Existing research has often focused on instrumental tasks like generating code or comments, rather than building foundational linguistic resources for studying the annotations themselves. The authors aim to bridge this gap by creating a large, well-curated corpus of these annotations, enabling deeper linguistic analysis.
Methods
The core of this paper is the development of a tool designed to automatically extract source code annotations. The process involves:
- Extraction: The tool pulls comments and changelogs (commit messages) from public, open-source code repositories.
- Tokenization and POS Tagging: Extracted text is automatically tokenized (broken into words) and part-of-speech (POS) tagged using NLTK.
- Sanitization: A significant part of the methodology is dedicated to cleaning the extracted data. This includes:
- Encoding and Character Cleaning: Resolving inconsistent encodings and removing invalid characters (like the bell character).
- Commented-Out Code Detection: Automatically identifying and discarding source code that has been commented out. For Python, this involves checking if the comment is valid Python code and contains specific characters or keywords (parentheses, brackets, equals signs, periods, "return"). This is a crucial step to prevent the corpus from being polluted with actual code.
- Snippet Handling (Future Work): While not fully implemented, the authors acknowledge the need to identify and preprocess snippets of actual code embedded within natural language comments.
The data is structured and stored in XML format, with each annotation represented as a <note> element containing raw text, tokens, POS tags, and metadata like author, repository, revision, and note type.
Datasets
The authors created a corpus of approximately 6 million words, comprising 90,461 changelogs and 76,445 comments. This corpus was extracted from three well-established, open-source repositories:
- Python:
django - C:
libvirt - C++:
dlib
This newly introduced benchmark is notable for its scale and the inclusion of multiple annotation types across different programming languages, differentiating it from prior, more limited efforts.
Key Insights and Ideas
- The "Messiness" of Code Comments: The authors highlight that code comments are a "very messy domain," requiring substantial sanitization. This underscores the practical challenges in automatically processing natural language embedded within code.
- The Trade-off Between Precision and Recall: When discarding commented-out code, the authors prioritize precision (ensuring no actual code is included) over recall (capturing all potentially useful comments). This means some natural language comments that coincidentally resemble code might be discarded, a trade-off they deem acceptable to maintain corpus integrity.
- The Linguistic Value of Annotations: The paper emphasizes the potential of source code annotations to reveal how natural language interacts with formal languages, and how this interaction influences linguistic choices. This opens avenues for studying language evolution and adaptation in specialized domains.
- The Need for Specialized Tools: The challenges in identifying commented-out code and handling embedded code snippets demonstrate that generic NLP tools may not be sufficient for analyzing code annotations, necessitating domain-specific approaches.
Key Results
- Corpus Creation: A 6-million-word corpus of English-language comments and changelogs was successfully constructed from three major open-source projects in Python, C, and C++.
- Commented-Out Code Detection: The implemented Python-specific rules for detecting commented-out code achieved a precision of 1.0 and a recall of 0.56 on a random sample, indicating high accuracy in identifying actual code while missing some comments.
- Feasibility Demonstration: The project demonstrates that building large-scale corpora of source code annotations is feasible, even with the inherent complexities of the data.
Limitations and Future Work
- Commented-Out Code Detection Limitations: The current automated detection of commented-out code is only fully functional for Python. Extending this to C/C++ is a significant challenge due to the nature of the
libclangparser. - Code Snippet Handling: Identifying and preprocessing embedded code snippets within natural language comments is planned for a future version. This is noted as a more challenging task than discarding commented-out code due to the need for higher accuracy.
- Diachronic Analysis: The current tool takes a snapshot of repositories. Future work could involve tracking repositories across multiple revisions to analyze how comments evolve over time.
- Documentation Extraction: The tool currently does not specifically extract formal documentation (like Doxygen), although documentation encoded as comments is included.
- Learned Models: The authors suggest that augmenting symbolic rules with learned models could help improve the accuracy of identifying commented-out code and potentially reduce the over-discarding of natural language comments.
Summary updated