Graph Neural Networks (GNNs) have become the standard for modeling relational data, but they suffer from a fundamental deployment challenge: inference latency. Because GNNs rely on recursive neighborhood aggregation, they require the entire graph structure to be present at runtime and involve complex data movements that scale poorly with graph size.
In response, a new research direction has emerged that seeks to "distill" the relational wisdom of GNNs into simple Multi-Layer Perceptrons (MLPs). By training MLPs to internalize graph topology during training, researchers are creating models that achieve GNN-level accuracy while running up to 800x faster during inference.
The Structural Blindness of MLPs
The primary reason standard MLPs fail on graph tasks is their assumption that data points are Independent and Identically Distributed (IID). As noted by Zhichun Guo and Nitesh Chawla in Linkless Link Prediction via Relational Distillation, MLPs process node features in isolation, effectively ignoring the edges that define the network's behavior.
GNNs overcome this by explicitly aggregating features from neighbors. However, this "structural awareness" comes at a high cost. At inference time, a GNN must fetch the features of a node’s neighbors, and their neighbors' neighbors, creating a massive computational overhead known as the "neighbor explosion" problem.
Relational Distillation: Teaching the MLP to See
The core insight of this research area is that the graph structure can be treated as a "privileged" signal—available during training but absent during inference. Through Knowledge Distillation, a complex "teacher" GNN can transfer its understanding of the graph to a "student" MLP.
In the Linkless Link Prediction (LLP) framework, the MLP is trained to mimic the output distributions and ranking patterns of a GNN. Instead of just learning to predict a label, the MLP learns the relational context that the GNN discovered. Once trained, the MLP can perform tasks like link prediction using only node features, achieving a 70x speedup because it no longer needs to traverse the graph.
Mechanisms for Injecting Topology
To make this distillation effective, the MLP needs more than just a teacher's output; it needs inductive biases that help it represent structural information. Several mechanisms have been proposed to bridge this gap:
- Positional Encodings and Adversarial Training: In NOSMOG, Chuxu Zhang and colleagues use positional encodings to give the MLP a sense of "where" a node is in the graph. Combined with representational similarity distillation, this allows the MLP to achieve up to 833x faster inference than traditional GNNs.
- Graph Vector-Quantization: Ling Yang and Minkai Xu introduced VQGraph, which uses a structure-aware tokenizer. This tokenizer maps complex local neighborhoods into a discrete "codebook," allowing the MLP to learn rich structural patterns as if they were simple feature tokens.
- Self-Distillation and Contrasting: Rather than relying solely on a separate teacher, Jun Xia proposed GSDN, where the MLP performs self-distillation on its own neighborhood representations. Similarly, Lirong Wu developed GSSC, which uses self-contrasting to decouple node features from topology during the forward pass, making the model more robust to graph noise.
Why It Works: Implicit Structural Mapping
The reason an MLP can eventually outperform or match a GNN is that the distillation process forces the MLP’s weights to act as a latent map of the graph.
In a standard GNN, the graph structure is an input. In a distilled MLP, the graph structure is baked into the parameters. As Yunhui Liu demonstrates in the HG2M framework for heterogeneous graphs, the MLP learns to transform raw features into a space where "relational semantics" are already resolved.
This shift from explicit aggregation to implicit mapping allows these models to handle massive, real-time industrial applications—like social media recommendations or fraud detection—where the latency of a traditional GNN would be prohibitive. By moving the structural complexity from the inference engine to the training objective, this research direction provides a path toward sample-efficient, lightning-fast agents that understand the world's connections without being slowed down by them.