Traditional deep learning models for temporal data, such as Recurrent Neural Networks (RNNs) or Transformers, treat time as a sequence of discrete steps. They assume that an observation at time $t$ leads directly to a state at $t+1$. However, real-world processes—from the progression of a disease to the turbulence of a fluid—evolve continuously.
The research direction of Neural Differential Equations shifts the paradigm from learning discrete transitions to learning the underlying "velocity field" of a system. By parameterizing the derivative of a latent state with a neural network, these models can evolve representations over any arbitrary time interval, naturally handling irregular sampling and enabling long-range extrapolation.
The Intuition of the Velocity Field
In a standard network, we learn a mapping $h_{t+1} = f(h_t)$. In a Neural Ordinary Differential Equation (Neural ODE), we instead learn the rate of change: $\frac{dh(t)}{dt} = f(h(t), t, \theta)$.
As Daniel Wang and Alex Wong demonstrate in ODE-GS: Latent ODEs for Dynamic Scene Extrapolation with 3D Gaussian Splatting, this approach treats the evolution of a system as a smooth flow. Instead of "jumping" between frames in a video or snapshots of a 3D scene, the model learns a continuous vector field. To find the state of the system at any future time $T$, we simply call an ODE solver to integrate the learned derivative from the initial state. This acts as a natural smoothness prior, ensuring that predicted future states follow physically plausible paths rather than the jittery transitions common in autoregressive models.
Bridging the Gaps in Irregular Data
One of the most significant advantages of this continuous framing is its ability to handle "gappy" or irregularly sampled data. In clinical settings, patients are not monitored at perfectly even intervals; one might have check-ups three days apart, then three months apart.
Barak A. Pearlmutter and Mansura Habiba address this in Neural ODEs for Informative Missingness in Multivariate Time Series by treating hidden state dynamics as a continuous flow. Similarly, the MIRA foundation model by Hao Li uses Neural ODEs within a Transformer architecture to extrapolate latent states to arbitrary future timestamps. Because the model understands the continuous trajectory, it can "fill in" the missing information between sporadic observations more accurately than discrete models that require fixed-interval binning.
Incorporating Physics and Stochasticity
When the system being modeled has known physical constraints or inherent randomness, Neural ODEs can be extended into "gray-box" models.
- Physical Constraints: In Finite Volume Neural Network: Modeling Subsurface Contaminant Transport, researchers embed spatial flux calculations directly into the ODE framework. This ensures that the learned dynamics obey conservation laws, allowing the model to generalize to unseen boundary conditions.
- Stochasticity: Real-world systems are rarely perfectly deterministic. Patrick Kidger explores this in Neural SDEs as Infinite-Dimensional GANs, using Stochastic Differential Equations (SDEs) to model latent trajectories. By parameterizing a diffusion term, the model learns to characterize the structure and magnitude of system noise, which is critical for high-volatility environments like finance or robotics.
Beyond Time Series: Continuous Representations
The utility of continuous latent dynamics extends beyond traditional time-series forecasting. It is increasingly used to provide structure to high-dimensional, unstructured data:
- 3D Point Clouds: Davis Rempe and Leonidas Guibas introduced CaSPR, which maps raw point clouds into a "4D container." By solving a differential equation in a low-dimensional latent space, they can reconstruct continuous object shapes and poses regardless of the original sampling frequency.
- Large Language Models: Even in the discrete world of tokens, continuity offers insights. HD-NDEs models the generation process of an LLM as a continuous trajectory through latent space. By analyzing the "flow" of information across a sequence, the model can detect hallucinations that might be invisible when looking at individual, isolated token states.
By moving from discrete steps to continuous flows, this research direction provides a mathematically rigorous way to model the "connective tissue" between observations, making AI systems more robust to the irregular and noisy nature of the physical world.