Continuous-Depth Dynamics for Graphs and Sequences

This direction is merged into another direction: neural ordinary differential equations (continuous-time latent dynamics). Its last version stays readable and citable here.

Version 1 of 121 papers22 conceptsfrozen 2026-10-04

In traditional deep learning, we think of data moving through a sequence of discrete layers: $h_{n+1} = f(h_n)$. However, many real-world systems—from the spread of rumors on social networks to the physical flow of traffic—evolve continuously. Recent research has shifted toward viewing neural networks as continuous-time dynamical systems, where the change in a state is governed by an Ordinary Differential Equation (ODE): $\frac{dh}{dt} = f(h(t), t, \theta)$.

By treating "depth" as "time," researchers are building architectures that are more flexible, physically grounded, and capable of handling the "messy" irregularities of real-world data.

Bridging the Gap in Irregular Time Series

Standard models like Transformers or RNNs struggle with irregularly sampled data because they assume a fixed interval between observations. If a patient’s heart rate is recorded at 2:00 PM and then again at 2:05 PM, a discrete model treats this the same as a five-hour gap.

Dongsheng Li, Kan Ren, and their colleagues addressed this with ContiFormer (ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling). By embedding Neural ODEs directly into the attention mechanism, ContiFormer allows the model to "query" the state of a system at any arbitrary point in time. Instead of jumping between discrete tokens, the model follows a continuous trajectory, capturing the underlying dynamics that persist even when data is missing.

Similarly, in the domain of social media cascades, Xin Jing 0003 and Dingqi Yang developed CasFT (CasFT: Future Trend Modeling for Information Popularity Prediction with Dynamic Cues-Driven Diffusion Models). They use Neural ODEs to model the "invisible" growth trends of information popularity, treating the viral spread of a post as a continuous evolution rather than a series of static snapshots.

Solving the Oversmoothing Problem with Physics

In Graph Neural Networks (GNNs), a common failure mode is oversmoothing: as you add more layers, the features of all nodes begin to converge, making them indistinguishable. Yulong Yang and Christine Allen-Blanchette recently demonstrated that this phenomenon is mathematically equivalent to reaching a "consensus" in social opinion dynamics (Understanding Oversmoothing in GNNs as Consensus in Opinion Dynamics).

To counter this, researchers are injecting physical "biases" into the ODEs:

  1. Hamiltonian Dynamics: Simon Heilig and Davide Bacciu introduced PH-DGN (Injecting Hamiltonian Architectural Bias into Deep Graph Networks for Long-Range Propagation). By modeling message-passing as a port-Hamiltonian system, they create a framework where information is conserved rather than dissipated. This allows for networks with thousands of layers where information can travel across massive graphs without decaying.
  2. Signed Interactions: Real-world graphs often have "negative" edges (e.g., distrust in a social network or inhibition in a gene circuit). Lanlan Chen and Jian Lou developed SGODE (Signed Graph Neural Ordinary Differential Equation for Modeling Continuous-time Dynamics) to handle these signed interactions. By incorporating social balance theory into the ODE, the model prevents the "averaging out" of conflicting signals, maintaining the distinct identities of nodes even in deep architectures.

The Numerical Engine: Stability and Solvers

The power of these models comes from the numerical solvers (like Euler or Runge-Kutta methods) that integrate the differential equations. However, this "engine" is also a source of fragility.

As noted by Linhao Luo and Shirui Pan in their work on GSNOP (Graph Sequential Neural ODE Process for Link Prediction on Dynamic and Sparse Graphs), these solvers are highly sensitive to hyperparameters like the learning rate and integration step size. If the step size is too large, the system can become unstable, leading to "gradient explosions" or erratic trajectories.

This sensitivity is a recurring theme across the literature. Whether it is Shibo Li and Shandian Zhe using ODEs to interpolate between simulation fidelities (Infinite-Fidelity Coregionalization for Physical Simulation) or Haiting Sun modeling dynamic signed networks (DynamiSE: dynamic signed network embedding for link prediction), the challenge remains the same: balancing the mathematical elegance of continuous time with the numerical realities of discrete computation.

Why It Works: The Intuition of Flow

The shift from discrete layers to continuous ODEs works because it changes the fundamental inductive bias of the network. Instead of learning a static mapping, the network learns the laws of motion for the data.

When we model a graph as a dynamical system, we aren't just stacking filters; we are simulating how features "flow" across edges. When we model a time series as an ODE, we aren't just predicting the next token; we are uncovering the latent forces driving the system forward. This perspective, as summarized by Lianfang Wang and Jun Liu, provides a mathematically grounded path toward more interpretable and robust AI (Deep Neural Networks Inspired by Differential Equations).

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_depth_dynamics_for_graphs_and_0ba356fd,
  title = {Continuous-Depth Dynamics for Graphs and Sequences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-depth-dynamics-for-graphs-and-sequences/rdr_0ba356fd95b84a59bab9c1681422bd50},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_continuous_depth_dynamics_for_graphs_and_0ba356fd_v1,
  title = {Continuous-Depth Dynamics for Graphs and Sequences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-depth-dynamics-for-graphs-and-sequences/rdr_0ba356fd95b84a59bab9c1681422bd50/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}