Empirical study of transformers for source code

Nadezhda Chirkova, Sergey Troshin

TL;DR

This paper presents a comprehensive empirical study on how Transformers can best leverage the syntactic structure of source code, represented as Abstract Syntax Trees (ASTs). The authors find that Sequential Relative Attention (SRA) is the most effective and efficient mechanism for capturing code structure, and they demonstrate that Transformers can make surprisingly accurate predictions using only syntactic information, even when variable names are completely anonymized.

What Problem Does This Paper Address?

While Transformers have revolutionized Natural Language Processing (NLP), applying them to source code isn't straightforward. Unlike natural language, code has a rigid, hierarchical structure defined by its syntax. Researchers have proposed various "AST-aware" Transformer modifications to capture this hierarchy, but these methods were often developed in isolation, tested on different tasks, and compared against weak baselines.

The authors set out to answer three key questions:

  1. What is the most effective way to feed AST structure into a Transformer?
  2. Can Transformers learn meaningful patterns from syntax alone (without relying on descriptive variable names)?
  3. Which parts of the AST (structure, node types, or values) are most critical for different software engineering tasks?

Methods

The authors re-implemented several state-of-the-art mechanisms in a unified framework to ensure a fair "apples-to-apples" comparison:

  • Sequential Positional Encodings: The standard NLP approach, treating the AST as a flattened sequence (depth-first traversal).
  • Sequential Relative Attention (SRA): Learns embeddings for the relative distance between tokens in a sequence.
  • Tree Positional Encodings: Encodes the path from the root to each node.
  • Tree Relative Attention: Learns embeddings based on the shortest path between nodes in the tree (e.g., "3 steps up, 2 steps down").
  • GGNN Sandwich: Alternates Transformer layers with Graph Gated Neural Network (GGNN) layers to combine global attention with local message passing.
Overview of different mechanisms for processing AST structure in Transformers.

A novel part of their methodology is the anonymized setting. They replaced all user-defined identifiers with generic placeholders (like var1, var2). This strips away the semantic "hints" in variable names, forcing the model to rely purely on the code's syntactic structure.

Datasets

The study used two large-scale, standard benchmarks:

  • Python150k (PY): 150,000 Python files from GitHub.
  • JavaScript150k (JS): 150,000 JavaScript files from GitHub.

Crucially, the authors released a new, cleaned split of these datasets. They followed best practices by splitting by repository (to prevent "data leakage" where files from the same project appear in both training and testing) and removing duplicate files.

Key Insights and Ideas

  • SRA is the "Sweet Spot": Despite its simplicity, Sequential Relative Attention (SRA) consistently performed as well as or better than more complex tree-specific mechanisms. It is fast, memory-efficient, and easy to implement.
  • Syntax is Sufficient: In the anonymized setting, Transformers still achieved high accuracy. This proves that the models are truly learning the "grammar" of code, not just memorizing common variable name patterns.
  • Task-Specific Needs:
    • Variable Misuse & Code Completion: These tasks rely heavily on all AST components—structure, node types, and values.
  • Function Naming: Surprisingly, this task relies almost entirely on node types and identifier names. The actual tree structure (how those nodes are connected) contributed very little to the model's performance.
  • The Power of Ensembling: Combining a model trained on full data with one trained on anonymized data led to significant performance boosts, suggesting these models capture complementary types of information.

Key Results

  • Variable Misuse: SRA achieved a joint localization and repair accuracy of 81.42% on Python, significantly outperforming standard positional encodings (~74.4%).
  • Code Completion: For value prediction in Python, SRA reached an MRR (Mean Reciprocal Rank) of 54.53%.
  • Anonymization Benefit: In some cases, like Python code completion, the anonymized model actually outperformed the full-data model. This is likely because anonymization helps the model handle "out-of-vocabulary" rare variable names more effectively by treating them as generic placeholders.

Limitations and Future Work

The authors focused purely on syntactic information. They did not incorporate data-flow (how data moves through variables) or control-flow (the order of execution) edges, which are common in other graph-based code models.

Future work could explore:

  • Combining SRA with subtokenization (splitting longVariableName into long, Variable, Name) to further improve vocabulary handling.
  • Investigating why Transformers struggle to utilize tree structure in the function naming task.
  • Applying these insights to pre-trained models like BERT or RoBERTa, which are currently the state-of-the-art for many code tasks but often treat code as simple text.

Summary updated

Abstract

Initially developed for natural language processing (NLP), Transformers are now widely used for source code processing, due to the format similarity between source code and text. In contrast to natural language, source code is strictly structured, i.e., it follows the syntax of the programming language. Several recent works develop Transformer modifications for capturing syntactic information in source code. The drawback of these works is that they do not compare to each other and consider different tasks. In this work, we conduct a thorough empirical study of the capabilities of Transformers to utilize syntactic information in different tasks. We consider three tasks (code completion, function naming and bug fixing) and re-implement different syntax-capturing modifications in a unified framework. We show that Transformers are able to make meaningful predictions based purely on syntactic information and underline the best practices of taking the syntactic information into account for improving the performance of the model.
BibTeX
@inproceedings{chirkova2021empirical,
  author = {Nadezhda Chirkova and Sergey Troshin},
  title = {{Empirical study of transformers for source code}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2021},
  doi = {10.1145/3468264.3468611},
}

Loading related content...

Figure