Abstract
Neural Ordinary Differential Equations (nODEs) offer an elegant, continuous-time framework for modeling dynamical systems, yet they often struggle with training stability, expressivity in low dimensions, and a lack of mechanistic interpretability. This paper introduces Gated Neural ODEs (gnODEs), a novel architecture that integrates the adaptive gating mechanisms of Recurrent Neural Networks (RNNs) into the nODE framework. By employing a novel critical initialization and a leak term, gnODEs overcome the Exploding and Vanishing Gradient Problem (EVGP). Our results demonstrate that gnODEs outperform standard nODEs and gated RNNs across diverse benchmarks, while naturally learning low-dimensional continuous attractors that provide a clear window into the model's internal computations.
1. Introduction
Neural Ordinary Differential Equations (nODEs) represent a paradigm shift in sequence modeling, treating the evolution of hidden states as a continuous flow. While theoretically appealing for their interpretability and ability to handle irregularly sampled data, nODEs face significant practical hurdles. Specifically, they are prone to the Exploding and Vanishing Gradient Problem (EVGP), making it difficult to learn long-term dependencies. Furthermore, the relationship between the architecture of the "velocity field" and the model's expressivity remains poorly understood.
To address these challenges, we propose the Gated Neural ODE (gnODE). By drawing inspiration from the gating mechanisms of GRUs and LSTMs, gnODEs introduce adaptive timescales that allow the model to modulate its dynamics dynamically. This architectural shift, combined with principled initialization strategies, significantly enhances both the trainability and the representational power of continuous-time models.
2. The Gated Neural ODE (gnODE) Framework
The core of the gnODE is defined by the following differential equation:
$$\tau \dot{h} = G_\phi(h, x) \odot [-h + F_\theta(h, x)]$$
Where:
- $G_\phi(h, x)$ (Gating): A neural network that computes an adaptive timescale. This allows the system to "slow down" to preserve memory or "speed up" to respond to rapid input changes.
- $-h$ (Leak Term): A linear decay term that ensures the system can be initialized in a stable or critical regime, preventing runaway dynamics.
- $F_\theta(h, x)$ (Flow Field): A deep neural network that defines the target state or "velocity" toward which the system evolves.
2.1 Critical Initialization
A major contribution of this work is a novel initialization scheme derived from dynamical mean-field theory. By placing the gnODE at the "edge of chaos" at the start of training, we ensure that gradients can propagate through long time horizons without vanishing or exploding. This addresses the primary bottleneck in training vanilla nODEs on complex temporal tasks.
2.2 Measuring Expressivity
To quantify the model's capacity, we utilize a metric inspired by "Gardner capacity" from statistical mechanics. This allows us to measure how effectively the network can fit stochastic trajectories (such as those generated by an Ornstein-Uhlenbeck process), providing a principled way to compare different architectures.
3. Experimental Evaluation
We evaluated gnODEs against standard nODEs and state-of-the-art gated RNNs (like LSTMs and GRUs) across a variety of synthetic and real-world tasks.
3.1 Synthetic Memory Probes
In the N-Bit Flip-Flop and Variable-Amplitude Flip-Flop tasks, models must store discrete or continuous values over time. While vanilla nODEs and RNNs failed to solve the variable-amplitude task in low dimensions ($N=6$), the gnODE solved it with high precision. This suggests that the gating mechanism provides a superior inductive bias for maintaining stable analog memory.
3.2 Real-World Benchmarks
- Walker2D: In predicting the kinematics of a bipedal robot, the gnODE achieved a Test MSE of 0.552 (at $N=32$), significantly outperforming standard nODEs (0.747) and the Long Expressive Memory (LEM) model (1.149).
- Speech Commands: On the task of classifying spoken words from audio, the gnODE reached 84.4% accuracy, surpassing GRUs and LSTMs across various hidden state sizes.
- CharacterTrajectories: The model demonstrated robust performance on irregularly sampled data, successfully classifying Latin alphabet characters from pen-tip trajectories.
4. Discussion: Interpretability and Continuous Attractors
One of the most significant findings is the gnODE's ability to learn continuous attractors. In the variable-amplitude flip-flop task, a 2D gnODE naturally organizes its latent space into a manifold of fixed points (e.g., a square or a line). To store a specific continuous value, the model moves the state to a specific point on this manifold, where it remains stable until a new input is received.
This behavior is highly interpretable; by plotting the flow fields in low dimensions, we can visually confirm how the network implements memory. Gated models rely on "marginally stable" fixed points (where eigenvalues are near zero), whereas non-gated models often fall into stable fixed points that cause them to "forget" information too quickly. This geometric insight explains why gating is essential for both performance and the emergence of robust memory structures.
5. Conclusion and Future Work
Gated Neural ODEs represent a significant step forward in continuous-time machine learning. By combining the stability of gated architectures with the elegance of differential equations, gnODEs achieve superior performance and interpretability.
While the current implementation uses Euler discretization for efficiency, future work will explore the integration of adaptive solvers to further enhance accuracy. Additionally, the parallels between gnODEs and biological systems—such as voltage-gated ion channels—suggest that these models may serve as a useful bridge between artificial intelligence and computational neuroscience.
Summary updated