Operator-Learning-Inspired Modeling of Neural Ordinary Differential Equations

Woojin Cho, Seunghyeon Cho, Hyundong Jin, Jinsung Jeon, Kookjin Lee, Sanghyun Hong, Dongeun Lee, Jonghyun Choi, Noseong Park

Introduction

Neural Ordinary Differential Equations (NODEs) represent a significant advancement in continuous-depth neural networks, defining the evolution of a hidden state $h(t)$ through an ODE: $dh(t)/dt = f(h(t), t)$. While conventional approaches model the function $f$ using standard layers like Fully Connected or Convolutional networks, this paper argues that $f$ is fundamentally a differential operator. Drawing inspiration from Scientific Machine Learning (SciML), the authors propose using Neural Operators—specifically a novel variant called the Branched Fourier Neural Operator (BFNO)—to model the ODE function, bridging the gap between operator learning and general-purpose deep learning.

Methods: The Branched Fourier Neural Operator

The core contribution is the Branched Fourier Neural Operator (BFNO). Standard Fourier Neural Operators (FNOs), while effective for solving partial differential equations, often lack the expressive power required for complex machine learning tasks like image classification. The BFNO architecture addresses this by introducing a dynamic global convolution mechanism.

The BFNO-NODE architecture models the ODE function $f$ as a sequence of operator layers.

A BFNO layer operates through the following steps:

  1. Spectral Transformation: The input is mapped to the frequency domain via a Fourier Transform.
  2. Dynamic Global Convolution: Instead of a single static filter, the model employs $L$ parallel global convolutional filters.
  3. Dynamic Aggregation: The outputs of these parallel branches are merged using a fully-connected layer, allowing the model to adaptively weight different spectral features.
  4. Spatial Recovery: The result is transformed back to the spatial domain, combined with a linear skip connection of the original input, and passed through an activation function.

This "branched" structure allows the network to maintain the mathematical rigor of neural operators while achieving the high expressivity needed for standard feature extraction.

Key Results and Insights

The BFNO-NODE architecture consistently outperformed existing NODE variants, such as ANODE, SONODE, and Heavy Ball NODEs, across image classification, time-series analysis, and generative modeling.

Test accuracy across various image datasets shows BFNO-NODE (red) significantly outperforming other NODE variants.

Performance Gains

In image classification, the advantages of BFNO were most pronounced on more challenging datasets. For instance, on STL-10, BFNO-NODE achieved an accuracy of 44.55%, a substantial lead over the next best baseline (ANODE at 37.11%). Similar trends were observed in time-series tasks, where the Latent-BFNO-NODE reached an accuracy of 0.874 on HumanActivity data. In generative modeling, the BFNO variant improved the Negative Log-Likelihood (NLL) on MNIST from 0.97 to 0.88.

Conceptual Shift

The success of BFNO-NODE suggests that treating the ODE function explicitly as a differential operator provides a more grounded and effective inductive bias for continuous-depth models. However, the authors noted that standard FNOs require structural modifications—like the parallel kernels in BFNO—to succeed in general ML contexts. Ablation studies highlighted that the number of parallel branches ($L$) is a critical hyperparameter; optimal performance was typically found at $L=2$ or $L=3$, with higher values leading to overfitting.

Conclusion

By integrating operator learning into the NODE framework, this work provides a more robust mathematical foundation for continuous-time modeling. While the reliance on Fast Fourier Transforms (FFTs) and the sensitivity of the branching hyperparameter present practical considerations, the significant performance improvements across classification and generative tasks mark a promising direction for the evolution of Neural ODEs. Future research may explore scaling these operator-based models to higher-resolution datasets and more complex physical simulations.

Summary updated

Abstract

Neural ordinary differential equations (NODEs), one of the most influential works of the differential equation-based deep learning, are to continuously generalize residual networks and opened a new field. They are currently utilized for various downstream tasks, e.g., image classification, time series classification, image generation, etc. Its key part is how to model the time-derivative of the hidden state, denoted dh(t)/dt. People have habitually used conventional neural network architectures, e.g., fully-connected layers followed by non-linear activations. In this paper, however, we present a neural operator-based method to define the time-derivative term. Neural operators were initially proposed to model the differential operator of partial differential equations (PDEs). Since the time-derivative of NODEs can be understood as a special type of the differential operator, our proposed method, called branched Fourier neural operator (BFNO), makes sense. In our experiments with general downstream tasks, our method significantly outperforms existing methods.
BibTeX
@inproceedings{cho2024operator,
  author = {Woojin Cho and Seunghyeon Cho and Hyundong Jin and Jinsung Jeon and Kookjin Lee and Sanghyun Hong and Dongeun Lee and Jonghyun Choi and Noseong Park},
  title = {{Operator-Learning-Inspired Modeling of Neural Ordinary Differential Equations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i10.29036},
}
Figure