Source code is a highly unusual artifact. Unlike natural language, which relies on fluid semantic context, or pure mathematical logic, which operates on abstract formal rules, code is inherently bimodal. It contains human intent—expressed through variable names, comments, and documentation—and machine instructions—structured by strict syntax trees and execution flows.
Early attempts to apply machine learning to source code typically picked a side. They either treated code as linear text, applying standard natural language processing (NLP) techniques, or treated it as a pure graph, applying graph neural networks (GNNs). However, as Song et al. note in their survey on automatic code comment generation, models that treat source code as simple linear text consistently underperform compared to hybrid models that integrate structural code analysis with sequence-to-sequence learning architectures. The core insight of modern machine learning for code (ML4Code) is that capturing the interplay between logic and naming conventions is essential for true program understanding.
The Multiple Modalities of Code
To understand why multimodal fusion works, we must first look at the distinct "views" or modalities that a single piece of code offers:
- The Lexical View: The raw sequence of tokens, identifiers, and comments. This stream captures developer intent. For instance, a variable named
is_authenticatedtells a model what the data means, even if the underlying logic is just a boolean flag. - The Syntactic View: The hierarchical structure of the code, typically represented as an Abstract Syntax Tree (AST). This defines the formal grammar of the language.
- The Control and Data Flow View: The runtime paths and dependencies. Control Flow Graphs (CFGs) map the execution paths (loops, branches), while Data Flow Graphs (DFGs) track how data moves between variables.
- The Dynamic/Execution View: Real-time execution metrics, such as memory usage, CPU registers, or compiler feedback.
Neglecting any of these views creates blind spots. As Wei et al. demonstrate in their work on supervised deep features for clone detection, lexical analysis misses structural changes, while syntactical analysis ignores the semantic context provided by variable and function names. Similarly, Rahman et al. show in their study of large Scala codebases that professional developers define code clones based on refactorability rather than syntax alone, requiring a multi-view representation that integrates identifier names with AST structures.
Mechanisms of Fusion
How do modern architectures combine these disparate modalities into a single, cohesive vector representation? Researchers have converged on three primary strategies.
Parallel Multi-Encoder Streams
The most intuitive approach is to process each modality through a specialized neural network architecture and fuse their outputs later. For example, a model might use a Transformer to process sequential text tokens and a GNN to process a control flow graph.
Lexical Tokens ---> [ Transformer Encoder ] ---\
---> [ Fusion Layer ] ---> Unified Embedding
Structural Graph ---> [ Graph Neural Network ] ---/
Wan et al. pioneered this with their Multi-modal Attention Network (MMAN) for source code retrieval, which simultaneously analyzes lexical tokens, ASTs, and CFGs through an attention-based fusion layer. By using specialized encoders, the model captures sequential, structural, and execution-path information to create a comprehensive semantic embedding.
In a similar vein, Li et al. proposed a dual-encoder architecture for code summarization that uses distinct neural networks to independently model raw code tokens and structured API sequences, capturing both low-level implementation details and high-level library interactions.
Multi-View Contrastive Alignment
Instead of fusing representations explicitly at the end of a network, contrastive learning forces different modalities of the same code snippet to align within a shared embedding space.
Wang et al. developed CODE-MVP, a pre-training framework that aligns multiple views of a program—including raw text, syntax trees, and execution graphs—using multi-view contrastive learning. By training the model to recognize that an AST and a text sequence represent the same underlying program, the model learns to map developer intent directly to execution logic.
Hybrid and Interleaved Encodings
Another strategy involves embedding structural information directly into a sequential stream, or vice versa, avoiding the need for separate encoders.
For instance, Bui et al. introduced InferCode, a self-supervised framework that combines structural node types from an AST with the actual text tokens found in the source code, training the model to predict subtrees.
In the domain of code generation and reasoning, Puerto et al. demonstrated that embedding natural language source text as comments within a generated code structure provides a logical scaffolding that helps language models navigate complex conditional reasoning.
Why Multimodal Fusion Resolves Ambiguity
To see why this works, consider the task of vulnerability detection. A pure language model might look at a function and conclude it is safe because the variable names look standard and the comments describe a benign operation. However, a structural model tracking data flow might notice that user input is passed directly to a system command without sanitization.
Ridoy addressed this by creating EnStack, an ensemble stacking framework that combines models specializing in structural representations (like data flow graphs) with those focused on syntactic-semantic understanding (like CodeBERT). The author demonstrated that these different architectural biases capture distinct, non-overlapping features of software vulnerabilities.
Similarly, Gao et al. developed DeepDFA, which guides efficient graph learning using dataflow analysis. By combining structural logic models with large language models, the authors created a hybrid system that benefits from both precise dataflow tracking and natural language context.
The same principle applies to hardware performance prediction. When mapping high-level code to hardware like FPGAs, Bai et al. showed in their work on the HLSYN benchmark that combining textual source code embeddings with structural graph-based representations allows models to capture both semantic patterns and the underlying control-data flow, outperforming unimodal models that focus exclusively on syntax or text.
Bridging Static Logic and Dynamic Execution
The ultimate frontier of multimodal code representation is bridging the gap between static code structure and dynamic execution behavior. Code is not static; its true meaning is realized when it runs.
To capture this, researchers are increasingly incorporating execution-side features. Liu et al. developed a representational strategy for personalized programming guidance that combines semantic code embeddings with execution-side features like memory usage, execution time, and compiler feedback to track student programming styles.
In high-performance computing, Teranishi et al. highlighted the necessity of training models on multiple representations of code—such as source text combined with intermediate representations and hardware performance counters—to accurately predict execution behavior and bridge the gap between high-level logic and low-level hardware utilization.
Even at the assembly level, Michi et al. utilized a hybrid neural architecture in AlphaDev that processes both the sequence of instructions using a Transformer and the dynamic CPU state (registers and memory) using a separate encoder, ensuring the reinforcement learning agent understands both code structure and its execution effects.
Key Foundational Work
Source Forager: A Search Engine for Similar Source Code (2017) — contributed 2 concept elements to this direction.
Using Machine Learning and Information Retrieval Techniques to Improve Software Maintainability (2012) — contributed 2 concept elements to this direction.
Supervised Deep Features for Software Functional Clone Detection by Exploiting Lexical and Syntactical Information in Source Code (2017) — contributed 2 concept elements to this direction.