In natural language processing, models typically operate on a fixed vocabulary of common words. However, source code presents a unique challenge: developers constantly invent new variable names, function identifiers, and class labels. As Hoa Khanh Dam et al. observed in A deep language model for software code, this creates a persistent "out-of-vocabulary" (OOV) problem where models encounter tokens they have never seen during training.
Unlike natural language, where the vocabulary size eventually plateaus, the number of unique identifiers in code grows almost linearly with the size of the codebase. Rafael-Michael Karampatsis et al. highlight in Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code that this unbounded growth makes traditional fixed-vocabulary models struggle to generalize, often replacing critical semantic information with generic "unknown" (UNK) tokens.
Breaking Identifiers into Subwords
The most common strategy for handling the open vocabulary problem is sub-tokenization. By breaking complex identifiers into smaller, recurring units, models can represent novel names as combinations of known pieces. Rafael-Michael Karampatsis et al. demonstrated that Byte-Pair Encoding (BPE) allows models to scale to massive datasets while virtually eliminating OOV errors.
However, standard BPE can sometimes fragment identifiers into semantically meaningless chunks. To address this, Jieke Shi et al. proposed a hybrid strategy in Can Identifier Splitting Improve Open-Vocabulary Language Model of Code? that combines BPE with heuristic splitting based on naming conventions like camelCase or snake_case. This ensures that the model preserves the semantic intent of the developer while maintaining the flexibility of an open vocabulary.
Copying from the Local Context
Even with sub-tokenization, a model might struggle to predict a project-specific identifier that appears for the first time in a file. To solve this, researchers use "copy" or "pointer" mechanisms. These allow a model to look at the surrounding code and "point" to an existing token to reuse it, rather than trying to generate it from scratch.
Srinivasan Iyer et al. utilized this in Summarizing Source Code using a Neural Attention Model to ensure that generated summaries correctly reference specific variable names. Similarly, Daniel Tarlow et al. showed in Learning to Fix Build Errors with Graph2Diff Neural Networks that pointer mechanisms are essential for automated code repair, as they allow the system to precisely reference local symbols that were never part of the global training set.
Preserving Logic through Anonymization
Another intuition for handling OOV identifiers is that the name of a variable often matters less than its role in the program's logic. Nadezhda Chirkova et al. explored this in A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code by "anonymizing" rare identifiers with placeholders like VAR1 or VAR2.
This approach works because it forces the model to focus on structural patterns—such as data flow and control structures—rather than getting distracted by unique naming conventions. By preserving the identity of variables (e.g., knowing that VAR1 on line 5 is the same as VAR1 on line 10) without needing to know their literal names, models can learn reusable edit patterns that generalize across different projects.
Structural and Type-Based Inference
Recent work has moved toward using the inherent structure of code to resolve OOV tokens. Marc Brockschmidt et al. argued in Generative Code Modeling with Graphs that Graph Neural Networks (GNNs) can capture the semantic consistency of identifiers by modeling them as nodes in a syntax tree.
Furthermore, Md Rizwan Parvez proposed in Building Language Models for Text with Named Entities that incorporating entity type information can help models predict rare identifiers. If a model knows that an OOV token is a List or a String, it can narrow down the likely operations performed on that token, even if the specific name is unknown. This shift from literal matching to structural and type-based reasoning is central to modern open-vocabulary learning in software engineering.
Key Foundational Work
A deep language model for software code (2016) — contributed 1 concept elements to this direction.
End-to-End Prediction of Buffer Overruns from Raw Source Code via Neural Memory Networks (2017) — contributed 1 concept elements to this direction.
Summarizing Source Code using a Neural Attention Model (2016) — contributed 1 concept elements to this direction.