Standard deep learning architectures operate on a fundamental assumption: data points or tokens can be processed as independent or fully connected entities. Multi-Layer Perceptrons (MLPs) treat inputs as independent and identically distributed (i.i.d.) vectors, while Transformers treat inputs as a flat sequence or set of tokens.
However, real-world data is rarely flat. When interconnected data—such as molecules, citation networks, or state-transition graphs—is fed into these standard architectures, a severe problem emerges: structural blindness. The models can see the individual features of each node, but they are completely blind to the topology, connectivity, and relational dependencies that bind them together.
The Cost of Topology-Blindness
To understand why structural blindness is fatal, imagine two entirely different chemical compounds made of the exact same atoms. A standard MLP processes each atom's features independently. Because it lacks what Chen et al. describe as an awareness of the surrounding graph topology in SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP, the MLP will generate identical representations for both compounds. It cannot "see" the edges. As Yao et al. demonstrate in MuGSI: Distilling GNNs with Multi-Granularity Structural Information for Graph Classification, without specialized mechanisms to account for geometry, models fail to distinguish between graphs that share identical feature distributions but possess completely different connectivity patterns.
This blindness forces models to waste immense computational capacity. In deep reinforcement learning, Li et al. observe that treating environment states as flat vectors forces an agent to spend millions of training cycles rediscovering basic connectivity that is already inherent in the environment's transition topology (Towards Sample Efficient Agents through Algorithmic Alignment (Student Abstract)).
Similarly, in software engineering, Yang et al. show that traditional test prioritization techniques fail to catch critical system bugs because they treat test inputs as independent, ignoring the complex topological interactions that trigger failures (Toward efficient testing of graph neural networks via test input prioritization).
While Graph Neural Networks (GNNs) solve this by using message-passing algorithms to aggregate neighbor features, they do so at a high cost. Message passing requires keeping the graph's adjacency matrix in memory, making GNNs slow, computationally expensive, and difficult to scale to massive datasets. As Zhang et al. note in Structure-Aware Group Discrimination with Adaptive-View Graph Encoder: A Fast Graph Contrastive Learning Framework, MLPs are highly efficient but graph-blind. The core challenge of this research direction is bridging this gap: how do we give fast architectures the structural awareness of a GNN without the computational overhead of explicit message passing?
Baking Structure into Fast Architectures
One of the most active solutions is to "bake" structural topology directly into the weights of an MLP during training, allowing the model to remain completely edge-free and lightning-fast during inference.
[Graph Data] ---> (GNN Teacher) --[Structural Knowledge]--+
| (Distillation)
v
[Node Features] ------------------------------------> (MLP Student) ---> [Structure-Aware Predictions]
Instead of passing messages along edges at runtime, models can learn to recognize local neighborhoods implicitly. For example, Wu et al. introduce a prototype-guided knowledge distillation framework (Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs) that transfers neighborhood context from a GNN teacher to an MLP student.
To make this representation space explicit, Yang et al. propose discretizing local graph structures into a searchable codebook of "structural tokens" in VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs. By learning to map node features to these structural tokens, an MLP can look up the geometric shape of its neighborhood without ever processing an edge.
This structural injection can even be achieved without a heavy GNN teacher. In A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation, the authors utilize dual self-distillation at both the feature and label levels to force an MLP to preserve structural topology. Similarly, Wu et al. use a self-contrasting mechanism (Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting) to decouple features from topology during the forward pass, training the MLP to implicitly predict structural relationships from raw features alone.
Restructuring Attention and Sequences
Structural blindness also plagues sequence-based models like Transformers and Large Language Models (LLMs). When a graph is flattened into text or a sequence of tokens, its structural properties are mangled. Global attention mechanisms treat the graph as a fully connected clique, ignoring specific structural constraints and edge types that define semantic meaning (Gurevych et al., Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs).
To cure this blindness, researchers modify the attention matrix to enforce topological constraints. For Directed Acyclic Graphs (DAGs), Luo et al. replace standard global attention with a reachability-based mechanism (Transformers over Directed Acyclic Graphs). By restricting a node's attention strictly to its ancestors and descendants, the model naturally mirrors the directional flow of the graph.
When predicting neural network performance, Yi et al. demonstrate that identifying a layer's operation type is useless without its connectivity context (such as the number of incoming connections), which they solve by grafting GNN inductive biases directly onto a Transformer architecture (NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning).
Standard Transformer Attention: Reachability-Based Attention (DAG):
A B C D A B C D
A [ X X X X ] A [ X . . . ] (A can only attend to self)
B [ X X X X ] B [ X X . . ] (B can attend to ancestor A)
C [ X X X X ] C [ X . X . ] (C can attend to ancestor A)
D [ X X X X ] D [ X X X X ] (D can attend to all ancestors)
For generative LLMs operating on complex networks, Jin et al. show that instead of flattening a network into a standard text prompt, allowing the model to execute explicit, iterative navigation steps across nodes and edges prevents hallucinations and preserves relational logic (Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs).
Beyond Pairwise Connections: Higher-Order Blindness
Intriguingly, the push to eliminate structural blindness has revealed that even standard GNNs suffer from a form of it. Because traditional message passing operates purely across pairwise edges, standard graph architectures are blind to higher-order topological features like multi-way loops, cycles, and voids.
As argued by Papamarkou et al. in Position: Topological Deep Learning is the New Frontier for Relational Learning, moving beyond graph-based pairwise constraints to higher-order structures like simplicial and cell complexes provides the mathematical framework needed to resolve this limitation. By building models that are explicitly aware of both local features and global topology, researchers are creating a generation of deep learning architectures that are no longer blind to the structure of the world they are trying to model.
Key Foundational Work
Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020) — contributed 1 concept elements to this direction.
Towards Sample Efficient Agents through Algorithmic Alignment (Student Abstract) (2021) — contributed 1 concept elements to this direction.
GAHNE: Graph-Aggregated Heterogeneous Network Embedding (2020) — contributed 1 concept elements to this direction.