Enhancing Neural Code Representation with Additional Context

Huy Nguyen, Christoph Treude, Patanamon ThongtanunamarXiv 2025 · DOI · arXiv

Abstract

Traditional deep learning models for software engineering tasks typically process code as isolated, static snapshots. This approach ignores the rich evolutionary history and structural relationships that human developers rely on for comprehension. This paper explores the integration of "additional context"—specifically version history, call graphs, and method age—into neural code representations. By evaluating five representative models (ASTNN, CodeBERT, GraphCodeBERT, CodeT5, and PLBART) across tasks such as clone detection, classification, and summarization, we demonstrate that contextual augmentation significantly improves performance. Notably, version history emerges as a powerful signal for identifying semantic clones, while call graphs substantially enhance code classification accuracy.

Introduction

When developers interact with a codebase, they rarely view a single method in isolation. Instead, they leverage version control systems to understand evolution and navigate call graphs to grasp system-wide dependencies. Conversely, state-of-the-art models like CodeBERT often lack this broader perspective. This research addresses the gap by investigating whether feeding models the raw source code of a method’s history and its structural neighbors can lead to more robust code understanding.

Methodology: Contextual Augmentation

The study introduces a framework to enrich neural models with three distinct contextual signals:

  1. Version History: The source code of previous iterations of a method.
  2. Call Graphs: The source code of the most significant caller and callee methods.
  3. Method Age: A numerical representation of the method's longevity in the repository.

To integrate this data, three fusion strategies were tested: Concatenation, Max-Pooling, and Diff-Concat. The latter is specifically designed for clone detection, calculating the absolute difference between two code snippets before appending contextual information.

Key Insights: The Value of Evolutionary Context

A primary finding of the study is that version history is the most consistent performance booster. It captures "evolutionary traces"—patterns of change that reveal a method's intent more clearly than any single version.

Figure 1: Comparison of current and historical code versions for clone detection. Historical versions often reveal commonalities that are obscured in current snapshots.

As illustrated in the figure above, current versions of code snippets (such as LongAdder and DoubleAdder) may appear distinct due to optimizations or specific data types. However, their historical versions often share variable names and logic structures, making them much easier for models to identify as semantic clones. This demonstrates that version history provides a bridge for models to recognize functional similarities that are not obvious in the final source code.

Quantitative Results

The inclusion of context led to substantial gains across all evaluated tasks:

  • Code Clone Detection: Utilizing version history improved CodeT5’s F1 score by 15.92%. The synergy of history, call graphs, and age resulted in gains up to 21.48%.
  • Code Classification: The ASTNN model saw a dramatic 25.84% increase in macro-F1 when augmented with call graph data.
  • Code Summarization: While automated metrics like BLEU showed modest improvements (e.g., GraphCodeBERT saw a 5.56% METEOR boost), human evaluators reported significantly higher scores for accuracy and adequacy in context-aware summaries.

Conclusion

Enriching neural code representations with version history and structural context provides a significant leap in model performance. While challenges remain—such as managing the limited context window of Transformer architectures and extending these techniques to other programming languages—the results clearly indicate that "looking beyond the snapshot" is essential for the next generation of AI-driven software engineering tools.

Summary updated

BibTeX
@misc{nguyen2025enhancing,
  author = {Huy Nguyen and Christoph Treude and Patanamon Thongtanunam},
  title = {{Enhancing Neural Code Representation with Additional Context}},
  year = {2025},
  eprint = {2510.12082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12082},
}
Figure