Standard Graph Neural Networks (GNNs) suffer from a phenomenon often described as the "heat death" of information. As these networks grow deeper, the message-passing process—which typically mimics first-order physical diffusion—causes node features to converge toward a uniform average. This "over-smoothing" renders the representations of different nodes indistinguishable, limiting GNNs to just a few layers.
To overcome this, a new class of architectures replaces the dissipative logic of heat diffusion with the energy-conserving logic of wave dynamics and oscillatory systems. By modeling information flow as a second-order process, these networks can maintain signal diversity across hundreds of layers.
The Problem with First Order Diffusion
Most GNNs are essentially discrete versions of the heat equation. In a heat system, gradients flow from high to low until everything reaches a uniform temperature. In a GNN, node features are "smoothed" by their neighbors in every layer. Mathematically, this is a first-order ordinary differential equation (ODE) where the rate of change depends only on the current state.
As Andi Han, Dai Shi, and their colleagues explain in From Continuous Dynamics to Graph Neural Networks: Neural Diffusion and Beyond, this first-order nature is the root of over-smoothing. Because the system naturally seeks equilibrium, the high-frequency information—the fine-grained differences that define individual nodes—is filtered out, leaving only a low-frequency "blur."
From Heat to Waves
The breakthrough in this field involves shifting to second-order dynamics, modeled after the wave equation. In a wave-like system, the change in a node's state depends not just on its current value, but also on its "velocity" (the rate of change from the previous layer).
In Graph-Coupled Oscillator Networks, T. Konstantin Rusch and Michael M. Bronstein demonstrate that this shift fundamentally changes the stability of the network. In a first-order system, the over-smoothed state is a stable equilibrium—the system wants to go there. In their second-order GraphCON framework, the over-smoothed state becomes an unstable equilibrium. The oscillatory nature of the system naturally pushes node features away from the uniform mean, preserving local information even as the network depth increases to 100 layers or more.
Preserving Identity through Synchronization
Another perspective on this problem comes from the study of coupled oscillators. Instead of nodes sharing feature values, they can be viewed as oscillators trying to synchronize their rhythms.
In From Coupled Oscillators to Graph Neural Networks: Reducing Over-smoothing via a Kuramoto Model-based Approach, Tan Minh Nguyen and Tuan Nguyen introduce KuramotoGNN. This model assigns each node a unique "natural frequency" based on its initial input features. While the nodes interact and synchronize their rates of change, their distinct natural frequencies act as a permanent anchor. This ensures that even if the nodes become "coordinated," they never become "identical," effectively halting feature collapse.
Similarly, Tingting Dan and Guorong Wu argue in Explore brain-inspired machine intelligence for connecting dots on graphs through holographic blueprint of oscillatory synchronization that this phase-based synchronization allows for complex "conversations" between data points without the loss of signal contrast inherent in traditional diffusion.
Energy Conservation and Long Range Propagation
The transition to wave dynamics also solves "over-squashing," where information from distant nodes fails to reach its destination. Because waves conserve energy rather than dissipating it, they can carry signals across long distances in a graph.
Alessandro Trenta and Alessio Gravina propose SONAR in SONAR: Long-Range Graph Propagation Through Information Waves. By using a second-order wave equation with adaptive resistance, SONAR allows information to propagate as oscillations. This maintains signal fidelity across the graph, ensuring that distant nodes can influence one another effectively without the signal dying out like heat in a cold room.
This principle extends beyond graphs. In Wavy Transformer, Satoshi Noguchi and Yoshinobu Kawahara apply these second-order dynamics to Transformer architectures. They show that the self-attention mechanism can be viewed as a diffusive process that leads to token uniformity. By replacing this with wave dynamics, the Wavy Transformer preserves feature diversity across deep layers, improving performance in NLP and vision tasks.
Why it Works: The Band-Pass Effect
The mathematical intuition behind these successes is that second-order systems act as band-pass filters. While first-order diffusion acts as a low-pass filter (removing high-frequency details), wave-like equations allow high-frequency signals to persist.
As noted in Second-Order Tensorial Partial Differential Equations on Graphs by Aref Einizade and Jhony H. Giraldo, these equations introduce inertia into the propagation. This inertia prevents the system from immediately jumping to a local average, allowing the network to capture periodic patterns and fine-grained details that are essential for complex tasks like traffic prediction or brain network analysis.