Learning Stochastic Dynamical Systems as an Implicit Regularization with Graph Neural Networks

Jin Guo, Ting Gao, Yufu Lan, Peng Zhang, Sikun Yang, Jinqiao DuanarXiv 2023 · arXiv · OpenReview

Abstract

Stochastic Gumbel graph networks are proposed to learn high-dimensional time series, where the observed dimensions are often spatially correlated. To that end, the observed randomness and spatial-correlations are captured by learning the drift and diffusion terms of the stochastic differential equation with a Gumbel matrix embedding, respectively. In particular, this novel framework enables us to investigate the implicit regularization effect of the noise terms in S-GGNs. We provide a theoretical guarantee for the proposed S-GGNs by deriving the difference between the two corresponding loss functions in a small neighborhood of weight. Then, we employ Kuramoto's model to generate data for comparing the spectral density from the Hessian Matrix of the two loss functions. Experimental results on real-world data demonstrate that S-GGNs exhibit superior convergence, robustness, and generalization compared with state-of-the-arts.

Introduction

Modeling high-dimensional, multivariate time series—such as brain signals (EEG) or financial market data—presents significant challenges due to spatial correlations and inherent noise. While Graph Neural Networks (GNNs) effectively capture spatial relationships, they often suffer from over-smoothing and poor generalization when deterministic models overfit training noise. Researchers have introduced Stochastic Gumbel Graph Networks (S-GGNs) to address these issues by treating multivariate time series prediction as a stochastic dynamical system.

Methodology: Stochastic Gumbel Graph Networks

The S-GGN framework represents system evolution using a discrete Stochastic Differential Equation (SDE):

$$X^{t+1} = X^t + f(X^t, A)\Delta t + \sigma(X^t, A)\xi_t \sqrt{\Delta t}$$

Key components include:

  • Drift Term ($f$): A GNN modeling deterministic dynamics.
  • Diffusion Term ($\sigma$): A GNN modeling stochasticity (noise) to capture system uncertainty.
  • Gumbel-Softmax Trick: Enables learning the latent graph structure ($A$) by sampling potential connections between variables.
  • Grouped Convolutions: Used in real-world scenarios to extract features and filter external noise before processing.

The authors mathematically demonstrate that the difference between the stochastic and deterministic loss functions acts as a regularization term dependent on the Hessian of the loss.

Implicit Regularization and Spectral Analysis

A central insight of this work is that the diffusion term ($\sigma$) provides implicit regularization, preventing the model from converging to sharp, narrow minima in the loss landscape. Spectral analysis of the Hessian matrix reveals that S-GGNs possess significantly smaller largest eigenvalues compared to deterministic GGNs. In deep learning theory, such 'flatter' minima are strongly correlated with improved generalization on unseen data.

Comparison of eigenvalue evolution over training epochs for GGN (left) and S-GGN (right).

Experimental Evaluation

The model was evaluated using Kuramoto oscillators and real-world wireless communication data. In the Kuramoto experiments, S-GGNs demonstrated a tighter concentration of eigenvalues in their spectral distribution, indicating more stable and efficient convergence.

Spectral density of the Hessian matrix for GGN (left) and S-GGN (right).

On wireless communication datasets, S-GGNs consistently outperformed standard GGNs, achieving lower Mean Squared Error (MSE) and Mean Absolute Error (MAE). Qualitative results further showed that S-GGN predictions track complex signal components more accurately than deterministic models, which often produce lagged or dampened results.

Conclusion

S-GGNs leverage stochasticity to improve the robustness and generalization of GNNs for multivariate time series. By incorporating a diffusion term that acts as an implicit regularizer, the framework achieves flatter loss landscapes and superior performance on noisy, non-linear data. Future work may explore more complex spatio-temporal datasets and the integration of sharpness-aware optimization techniques.

Summary updated

Abstract

Stochastic Gumbel graph networks are proposed to learn high-dimensional time series, where the observed dimensions are often spatially correlated. To that end, the observed randomness and spatial-correlations are captured by learning the drift and diffusion terms of the stochastic differential equation with a Gumble matrix embedding, respectively. In particular, this novel framework enables us to investigate the implicit regularization effect of the noise terms in S-GGNs. We provide a theoretical guarantee for the proposed S-GGNs by deriving the difference between the two corresponding loss functions in a small neighborhood of weight. Then, we employ Kuramoto's model to generate data for comparing the spectral density from the Hessian Matrix of the two loss functions. Experimental results on real-world data, demonstrate that S-GGNs exhibit superior convergence, robustness, and generalization, compared with state-of-the-arts.
BibTeX
@misc{guo2023learning,
  author = {Jin Guo and Ting Gao and Yufu Lan and Peng Zhang and Sikun Yang and Jinqiao Duan},
  title = {{Learning Stochastic Dynamical Systems as an Implicit Regularization with Graph Neural Networks}},
  year = {2023},
  eprint = {2307.06097},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.06097},
}
Figure