Abstract
We propose a neural operator framework, termed mixture density nonlinear manifold decoder (MD-NOMAD), for stochastic simulators. Our approach leverages an amalgamation of the pointwise operator learning neural architecture nonlinear manifold decoder (NOMAD) with mixture density-based methods to estimate conditional probability distributions for stochastic output functions. MD-NOMAD harnesses the ability of probabilistic mixture models to estimate complex probability and the high-dimensional scalability of pointwise neural operator NOMAD. We conduct empirical assessments on a wide array of stochastic ordinary and partial differential equations and present the corresponding results, which highlight the performance of the proposed framework.
Introduction
In science and engineering, Differential Equations (DEs) are fundamental for modeling physical phenomena. However, real-world systems are often stochastic, involving inherent randomness or uncertain parameters. Modeling these requires Stochastic Differential Equations (SDEs), which are computationally expensive to solve numerically. Traditional Uncertainty Propagation (UP)—understanding how input uncertainty affects the final result—typically necessitates thousands of Monte Carlo (MC) simulations. Even modern generative surrogate models, such as GANs or Diffusion models, act as sample generators that still require extensive statistical post-processing. MD-NOMAD bridges this gap by creating a surrogate that provides the entire probability density function (PDF) in a single forward pass, enabling "one-shot" uncertainty propagation.
Methods
MD-NOMAD (Mixture Density Nonlinear Manifold Decoder) merges two powerful concepts to achieve mesh-independent, analytical uncertainty estimation:
- NOMAD (Nonlinear Manifold Decoder): A neural operator architecture that uses a Branch Network to process input functions (initial conditions or parameters) and a Decoder Network for spatial/temporal coordinates. This allows the model to scale to high-dimensional problems and evaluate results at any arbitrary point.
- Mixture Density Networks (MDN): Rather than outputting a single scalar value, the decoder outputs the parameters (weights $\pi$, means $\mu$, and standard deviations $\sigma$) of a Gaussian Mixture Model (GMM).
By training with a Negative Log-Likelihood (NLL) loss, the model learns the conditional probability distribution of the system's response. Because the output is a GMM, the mean, variance, and full PDF can be calculated analytically, bypassing the need for Monte Carlo sampling during inference.
Key Results and Insights
The framework was evaluated across diverse benchmarks, including the Stochastic Van der Pol Equation, Lorenz-96 systems, and 2D Elliptic SPDEs.
- Handling Bifurcation: A standout capability of MD-NOMAD is capturing complex transitions where a distribution splits from one peak to two (bifurcation). As shown in the density plots, the model accurately tracks the evolution of multi-peaked distributions over time, a task traditionally difficult for standard surrogate models.
- Efficiency and Speed: In 2D SPDE tasks, MD-NOMAD achieved a speedup of approximately 750x compared to standard Monte Carlo/KDE approaches, computing PDFs in ~0.4 seconds versus ~301 seconds.
- Resolution Invariance: As a neural operator, the model can be trained on coarse grids and evaluated on finer or non-overlapping grids without retraining, avoiding the "curse of dimensionality."
- Comparison to Diffusion Models: MD-NOMAD was found to be 4.5x faster at inference than Denoising Diffusion Probabilistic Models (DDPM) while achieving an error rate an order of magnitude lower.
Conclusion
MD-NOMAD represents a significant advancement in stochastic emulation, providing a parametric approach to uncertainty propagation that is both faster and more accurate than generative alternatives. While currently utilizing univariate Gaussian mixtures, future work aims to incorporate multivariate distributions to capture inter-component correlations and scale the framework to 3D applications like turbulent channel flow. This framework holds promise for real-time reliability analysis in complex engineering structures.
Summary updated