In programming languages, developers enjoy symbolic freedom. The functional logic of a program is entirely decoupled from the arbitrary names assigned to its variables, functions, and classes. A compiler treats a variable named total_sum exactly the same as one named q3_x.
However, as neural network models have become the dominant tools for code analysis, search, and generation, researchers have uncovered a striking paradox: these models derive a massive portion of their performance not from understanding pure program logic, but from reading human-authored identifier names. When these descriptive labels are removed, randomized, or altered, the models' apparent comprehension frequently collapses. This phenomenon is known as semantic identifier dependency.
The Statistical Shortcut: Natural vs. Formal Channels
Source code contains two concurrent streams of information: the formal channel (the strict execution logic, control flow, and syntax rules) and the natural channel (human-readable identifiers and comments). Because deep learning models are typically trained on massive corpora of open-source code, they are exposed to strong statistical correlations between descriptive names and specific algorithmic behaviors.
Instead of learning the computationally demanding task of tracing data flows or simulating execution paths, models take a statistical shortcut. They treat identifier names as primary semantic anchors. Early structural models like code2vec: Learning Distributed Representations of Code by Uri Alon et al. and early generation systems like CODE-NN by Srinivasan Iyer et al. demonstrated that when code is obfuscated or uses non-descriptive identifiers, predictive performance degrades significantly. The models lose the textual scaffolding they rely on to interpret syntactic structures.
As Zhilong Wang et al. demonstrated in How Does Naming Affect Language Models on Code Analysis Tasks?, stripping or anonymizing identifiers can cause a model's performance on tasks like code search to plummet by approximately 75%. This reveals that code language models often prioritize literal, surface-level features over underlying logical structure and control flow.
Exposing Fragility via Adversarial Renaming
The depth of this dependency becomes obvious when models are subjected to adversarial perturbations. Because models over-rely on the natural channel, making minor, non-functional changes to human-readable elements can completely disrupt their predictions.
In Adversarial Examples for Models of Code, Noam Yefet et al. introduced a white-box attack (DAMP) that renames variables using gradient-based optimization. By changing just a few identifier names without altering program behavior, they successfully tricked models into making specific, incorrect predictions up to 89% of the time. Similarly, the black-box attack framework CodeAttack, developed by Akshita Jha in CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models, showed that state-of-the-art models like CodeT5 and CodeBERT are highly vulnerable to human-imperceptible modifications of variable names and operators.
# Original Code: Model easily understands intent
def calculate_area(radius):
return 3.14 * (radius ** 2)
# Adversarially Renamed Code: Model performance plummets
def download_file(payload):
return 3.14 * (payload ** 2)
This fragility is not limited to variable names; it extends to function signatures. Nuo Chen et al. found in Evaluating and Enhancing the Robustness of Code Pre-trained Models through Structure-Aware Adversarial Samples Generation that if a function signature is modified to contradict the internal code behavior, the model will prioritize the misleading signature over the actual operational logic.
This heuristic bias can even lead to inverse scaling in large language models. Antonio Valerio Miceli Barone et al. showed in The Larger they are, the Harder they Fail: Language Models do not Recognize Identifier Swaps in Python that when common built-in functions (like len and print) are explicitly redefined or swapped within a local scope, larger models fail more confidently than smaller ones. Their strong pre-trained lexical correlations override their ability to track scope and binding rules.
Quantifying the Semantic Gap
When models are forced to operate in environments stripped of natural language hints, their lack of deep structural understanding becomes apparent. This semantic gap has been quantified across several distinct dimensions:
- Counterfactual Analysis: In Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates, Ashish Hooda et al. utilized counterfactual mutations to show that replacing descriptive identifiers with random strings causes a substantial drop in success rates, proving that models rely on surface pattern matching rather than data flow relationships.
- Anonymized Benchmarks: Benchmarks like CLARC, introduced by Kaicheng Wang et al. in CLARC: C/C++ Benchmark for Robust Code Search, evaluate models on anonymized code and low-level assembly. They reveal that state-of-the-art embedding techniques experience severe performance drops when superficial lexical cues are removed.
- Compiler Intermediate Representations (IR): When evaluating models on LLVM IR, Hailong Jiang discovered in Can Large Language Models Understand Intermediate Representations in Compilers? that models consistently fail to reason about instruction-level logic and complex control flows, relying instead on high-level function names or common coding patterns.
- Structural Delimiters: The authors of Benchmarking Language Models for Code Syntax Understanding found that models rely heavily on explicit markers like reserved keywords or semicolons to infer code structure. When these surface-level anchors are absent, their ability to resolve logical dependencies drops significantly.
Breaking the Dependency: Grounding Models in Logic
To move past these superficial heuristics, current research focuses on forcing models to learn the underlying algorithmic logic of code independent of its naming conventions.
Obfuscation Grounding
One direct solution is to alter the pre-training distribution so that models cannot rely on lexical shortcuts. In ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding, Indraneil Paul et al. introduced a pre-training strategy where descriptive variable and function names are systematically replaced with generic placeholders. This forces the model to learn algorithmic logic from the ground up, yielding higher data efficiency and superior performance on complex reasoning tasks.
Contrastive and Template-Based Learning
Other approaches use specialized training objectives to enforce identifier invariance. The authors of ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning applied contrastive learning with data augmentations to explicitly teach models that a program's semantic meaning remains identical when identifiers are renamed.
To achieve true compositional generalization, Hojae Han et al. proposed CTBERT in Towards Compositional Generalization in Code Search. CTBERT automatically mines structural templates from Abstract Syntax Trees (ASTs), training the model to recognize structural idioms rather than relying on lexical pattern matching.
Code Erosion
For downstream tasks like code summarization, frameworks like VECOS, developed by Fan Yan et al. in Towards Generating Summaries for Lexically Confusing Code through Code Erosion, intentionally "erode" code into a standardized, identifier-neutral format. By using variational inference to align these eroded representations with their original functional meanings, the system ensures that summaries are generated based on structural logic rather than misleading or non-standard naming conventions.
Key Foundational Work
code2vec: Learning Distributed Representations of Code (2018) — contributed 2 concept elements to this direction.
Summarizing Source Code using a Neural Attention Model (2016) — contributed 1 concept elements to this direction.