Overcoming Structural Information Loss in Code Representations

794 concepts536 papers10 proposals

Standard natural language processing models treat text as a linear sequence of tokens. When applied to source code, large language models and sequential architectures (like RNNs or standard Transformers) adopt this same assumption: they read code from left to right, line by line.

However, code is fundamentally different from natural language. It is a mathematical graph masquerading as text. A program possesses strict hierarchical syntax, explicit execution paths, and complex data dependencies. When we flatten a codebase into a linear sequence of tokens, we introduce structural information loss. This structural blindness limits a model's ability to reason about deep program logic, track long-range variable states, or identify subtle security vulnerabilities.

To bridge this gap, a major research direction focuses on injecting structural topology—such as Abstract Syntax Trees (ASTs), Control Flow Graphs (CFGs), and Data Flow Graphs (DFGs)—directly into neural representations.

The Cost of Flatness

Why does sequential representation fail? Consider how a developer understands code. They do not merely read keywords; they track how data flows between variables and how control branches through loops and conditionals. Sequential models, by contrast, are prone to superficial pattern matching.

As Saikat Chakraborty et al. demonstrated, sequential deep learning models often achieve high accuracy on vulnerability detection benchmarks by merely memorizing dataset "artifacts" (like specific variable names or superficial vocabulary) rather than understanding actual logic flaws. Similarly, Hongyu Zhang et al. found that pre-trained models over-rely on identifiers and keywords, frequently failing to capture formal grammar and complex branching logic.

This sequential blindness manifests in three distinct ways:

  1. Hierarchical Disconnect: Code is deeply nested. In front-end development, for instance, maintaining the relationship between parent containers and child elements is vital. Shanchao Liang et al. noted that standard sequence-based attention mechanisms struggle with this nested hierarchy, leading to layout inconsistencies during code generation.
  2. Long-Range Blindness: A vulnerability or bug often depends on an interaction between two operations separated by hundreds of lines of text. In a flat sequence, these operations are far apart. As Xu Duan et al. argue, linear token sequences cannot naturally link these distant but logically dependent operations, making fine-grained security auditing exceptionally difficult.
  3. Functional Equivalence vs. Textual Divergence: Two code snippets can look identical but behave differently, or look completely different while performing the exact same function. Tianhao Peng et al. point out that raw text representations fail to account for this functional equivalence, preventing models from distinguishing between a superficial syntax error and a fundamental logical misconception.

Early attempts to use machine learning for code analysis relied on lexical features. For example, Fabian Yamaguchi utilized bags of API symbols to extrapolate vulnerabilities. However, as Fabian Yamaguchi noted, a bag of symbols completely misses conditional logic and validation checks; safe code and unsafe code look identical if they happen to call the same library functions.

Graph-Integrated Architectures

To prevent structural context loss, researchers began mapping source code and binaries to analysis-friendly graphs before feeding them into neural networks.

Instead of forcing a model to infer the rules of syntax and execution from raw text, these approaches explicitly provide the topology. A common target is the Code Property Graph (CPG), which fuses a program's AST, CFG, and DFG into a single unified representation.

Sequential Representation (Flat Token Stream):
[ def ] -> [ calculate ] -> [ ( ] -> [ x ] -> [ ) ] -> [ : ] ...

Graph-Integrated Representation (Multi-View Topology):
     [AST: Function Definition]
               |
       [CFG: Branch Condition]
        /                 \
[DFG: Variable X]   [DFG: Variable Y]

Graph Neural Networks (GNNs) are uniquely suited to operate on these structures because they propagate information along the actual execution and data pathways. For smart contract security—where minor logical oversights lead to catastrophic financial exploits—Yuan Zhuang et al. and Zhenguang Liu et al. utilized GNNs over normalized "contract graphs." By passing messages along data-flow and control-flow edges, their models successfully flag complex vulnerabilities like reentrancy and infinite loops that flat sequence models miss.

This graph-based paradigm extends down to machine code. When source code is unavailable, treating binary executables as simple sequences of text or n-grams strips away the underlying logic. To solve this, Aram Galstyan et al. developed Bin2vec, which transforms binary code into a structured program graph and processes it via Graph Convolutional Networks (GCNs), capturing high-level semantics directly from the structural flow of machine instructions.

Multi-View Fusion and Structure-Aware Attention

While pure graph models capture logic well, they can lose the rich semantic information embedded in natural language identifiers and comments. The field has therefore converged on multi-view fusion—combining textual token sequences with structural graphs.

The Multi-Modal Attention Network (MMAN) developed by Yao Wan et al. and the Multi-View Pre-Trained Model (MV-PTM) by Xuxiang Jiang et al. exemplify this. These architectures use parallel encoders to process the code text, its AST, and its execution paths simultaneously, fusing them through specialized attention layers.

Perhaps the most influential realization of this concept within Transformer architectures is GraphCodeBERT, introduced by Daya Guo et al.. Instead of radically altering the Transformer architecture, the authors injected structure directly into the pre-training phase by leveraging data flow.

GraphCodeBERT takes both code tokens and a graph of variable dependencies as input. It uses a graph-guided masked attention mechanism where a variable token is allowed to attend to its source or destination variables, regardless of how far apart they are in the source text. By forcing the model to predict where data flows, the model learns a representation rooted in how a program functions, drastically reducing syntactic hallucinations during code generation.

Standard Attention:   Token (i) attends to Token (i+1), Token (i+2)... based on proximity.
Structure Attention:  Variable Token attends directly to its initialization node via DFG edges, 
                      bypassing 50 lines of intermediate text.

This structural guidance can also mimic execution directly. David Bieber et al. introduced the Instruction Pointer Attention GNN (IPA-GNN), which executes programs "in-silico" by using a continuous attention mechanism to track a "soft" instruction pointer across a program's control flow graph, allowing the network to systematically generalize to unseen execution paths.

Scaling to Repositories and Projects

As these models move from single functions to real-world software engineering, the structural information loss compounds. Code in a modern project relies on a web of cross-file dependencies, class inheritances, and external APIs.

Standard retrieval-augmented generation (RAG) systems often fetch local code snippets using fixed token windows. As Yangruibo Ding et al. observed, this truncates syntax blocks and omits critical structural context. To mitigate the resulting API hallucinations, Mingyu Chen et al. introduced MARIN, which applies hierarchical project-wide static analysis to constrain the model's decoding process, ensuring generated code adheres to valid architectural boundaries.

Ultimately, the goal is to bake this repository-level topology directly into the model's core attention mechanism. The Code Graph Model (CGM) by Hongyuan Tao et al. achieves this by integrating a repository's structural dependencies directly into an LLM's attention layers. Instead of relying on complex, multi-step agent loops to search a codebase, the model's attention naturally follows architectural links like function calls and class inheritances across disparate files.

Why It Works: The Power of Inductive Biases

The transition from sequential code representation to structure-aware representation works because it introduces a powerful inductive bias.

A purely sequential model can theoretically learn the syntax and data-flow rules of a programming language, but it must do so entirely from statistical co-occurrences of text tokens. It has to waste parameter capacity reconstructing the implicit graph hidden within the text.

By explicitly providing the graph topology—whether through GNN message passing, multi-view fusion, or graph-guided attention matrices—we relieve the model of this reconstruction burden. The model no longer has to guess if a variable on line 5 affects an expression on line 105; the data-flow edge connects them directly. This shifts the neural network's task from inferring structure to reasoning over structure, leading to models that are more robust, less prone to memorizing dataset artifacts, and significantly more capable of understanding the true execution logic of software.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_structural_information_loss_sequential_c_c0b22198,
  title = {structural information loss (sequential code representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-information-loss-sequential-code-representation/rdr_c0b2219891f74e69b2917cf99870104e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_structural_information_loss_sequential_c_c0b22198_v1,
  title = {structural information loss (sequential code representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-information-loss-sequential-code-representation/rdr_c0b2219891f74e69b2917cf99870104e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake