Can Machines Read Coding Manuals Yet? - A Benchmark for Building Better Language Models for Code Understanding

Ibrahim Abdelaziz, Julian Dolby, Jamie P. McCusker, Kavitha Srinivas

  • A Benchmark for Building Better Language Models for Code Understanding

While AI has become remarkably proficient at generating code snippets, true "understanding" of software involves more than just syntactic correctness. A human developer doesn't just look at a script; they consult documentation, browse Stack Overflow, and understand how classes relate within a hierarchy. Most AI research has focused on the code itself, but a new study from researchers at IBM and RPI argues that we are ignoring a critical source of information: the textual artifacts surrounding code.

The Gap in Code Understanding

Most current "code understanding" models are trained on raw source code, often represented as Abstract Syntax Trees (ASTs) or data-flow graphs. While effective for tasks like autocompletion, these models often miss the high-level intent and functional relationships described in manuals or forum discussions.

For example, a model might see two different Python classes, GLM and SGDClassifier, and treat them as distinct entities. However, a human reading the documentation knows they both represent linear models and perform similar functions. Existing benchmarks like CodeXGLUE focus heavily on code-to-code or code-to-text translation, but they don't measure how well a model understands the relationships between textual descriptions of code. BLANCA fills this gap.

Introducing BLANCA

To address this, the authors introduce BLANCA (Benchmarks for LANguage models on Coding Artifacts), a suite of five tasks designed to evaluate how well language models understand natural language text about code. The tasks are built on GraphGen4Code, a massive knowledge graph linking 1.3 million Python programs to their corresponding documentation and Stack Overflow posts.

The BLANCA suite consists of five specialized tasks:

  • Forum Answer Ranking (R): Predicting which answer to a technical question is the most helpful.
  • Forum Link Prediction (L): Determining if two forum posts are semantically related.
  • Forum to Class Prediction (F): Matching a forum discussion to the specific code class or function it describes.
  • Class Hierarchy Distance Prediction (H): Predicting how far apart two classes are in a library's inheritance tree based on their documentation.
  • Class Usage Prediction (U): Determining if two classes are used in similar ways within actual codebases.

Methodology

The researchers evaluated several state-of-the-art language models, including general-purpose ones (BERT, RoBERTa) and domain-specific ones (CodeBERT, BERTOverflow). Using the Sentence-BERT (SBERT) framework, they mapped text into a vector space where semantically similar items are physically closer. They tested three scenarios: out-of-the-box performance, single-task fine-tuning, and multi-task training (RFLHU) across all BLANCA tasks simultaneously.

Key Results and Insights

Models were highly successful at distinguishing linked vs. unlinked posts. Fine-tuning significantly improved the separation between related and unrelated pairs in the embedding space.

Figure 2: Distribution of embedding distances for linked and non-linked forum post pairs across different models.

The T-statistic, which measures the separation between related and unrelated pairs, jumped from 59.5 for the base BERTOverflow to 198.1 for the multi-task version. This demonstrates that the models are learning a more nuanced semantic space for technical discussions.

Structural Understanding of Libraries

In the Class Hierarchy task, the correlation between documentation similarity and hierarchy distance improved significantly after fine-tuning. This proves that models can learn to "see" the structure of a library through its manual.

Figure 3: Correlation between pair embedding cosine distance and class hierarchy distance.

As shown in Figure 3, models like FT-BERTOverflow and HU-BERTOverflow show a clear linear relationship between the distance in the class hierarchy and the distance in the embedding space, whereas out-of-the-box models like CodeBERT (orange line) fail to capture this structural relationship.

Multi-task Synergy

The "RFLHU" models (trained on all tasks) often outperformed single-task models. This suggests that learning to rank forum answers actually helps a model understand class hierarchies, indicating a shared underlying "semantic space" for programming concepts. Interestingly, models trained on general English (like RoBERTa) perform reasonably well out of the box, suggesting that "coding language" is deeply rooted in natural language logic.

Conclusion

The BLANCA benchmark reveals that while general-purpose models are a decent starting point, fine-tuning on specific "text-about-code" tasks significantly improves a model's ability to grasp the deeper semantics of programming. By bridging the gap between code and its surrounding documentation, we can build "smarter" AI tools that understand not just what code does, but why and how it relates to the broader software ecosystem.

Summary updated

Abstract

Code understanding is an increasingly important application of Artificial Intelligence. A fundamental aspect of understanding code is understanding text about code, e.g., documentation and forum discussions. Pre-trained language models (e.g., BERT) are a popular approach for various NLP tasks, and there are now a variety of benchmarks, such as GLUE, to help improve the development of such models for natural language understanding. However, little is known about how well such models work on textual artifacts about code, and we are unaware of any systematic set of downstream tasks for such an evaluation. In this paper, we derive a set of benchmarks (BLANCA - Benchmarks for LANguage models on Coding Artifacts) that assess code understanding based on tasks such as predicting the best answer to a question in a forum post, finding related forum posts, or predicting classes related in a hierarchy from class documentation. We evaluate the performance of current state-of-the-art language models on these tasks and show that there is a significant improvement on each task from fine tuning. We also show that multi-task training over BLANCA tasks helps build better language models for code understanding.
BibTeX
@inproceedings{abdelaziz2022can,
  author = {Ibrahim Abdelaziz and Julian Dolby and Jamie P. McCusker and Kavitha Srinivas},
  title = {{Can Machines Read Coding Manuals Yet? - A Benchmark for Building Better Language Models for Code Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i4.20363},
}
Figure