Multimodal Scientific Learning Beyond Diffusions and Flows

Leonardo Ferreira Guilhoto, Akshat Kaushal, Paris PerdikarisarXiv 2026 · arXiv

TL;DR

While trendy diffusion and flow-based generative models are currently favored for handling complex distributions, this paper shows they are often overkill—and topologically flawed—for scientific machine learning (SciML) problems. By revisiting Mixture Density Networks (MDNs), the authors demonstrate that explicit parametric mixture models provide superior data efficiency, faster inference, and clearer physical interpretability when mapping multimodal physical systems.

What Problem Does This Paper Address?

In Scientific Machine Learning (SciML), we frequently encounter systems with multimodal conditional uncertainty. This happens when a single input can yield multiple valid physical outcomes—such as in ill-posed inverse problems, chaotic fluid dynamics, or systems with multiple stable states (multistability).

Standard neural networks trained with Mean Squared Error (MSE) fail catastrophically here because they only predict the conditional expectation (the average), which often lands in a physically impossible, low-probability region between the true modes. To fix this, researchers have recently turned to implicit generative models like diffusion and conditional flow matching (CFM). However, these models are notoriously data-hungry, computationally expensive at inference time, and struggle to accurately capture the relative weights of well-separated, disconnected physical states.

Methods

The authors formalize a framework contrasting implicit distribution networks (like diffusion and CFM) with explicit distribution networks like Mixture Density Networks (MDNs).

Instead of generating samples through an iterative denoising or transport process, an MDN directly predicts the closed-form parameters of a conditional probability density function. Specifically, it models the output as a finite mixture of Gaussian distributions:

$$p_\theta(y|x) = \sum_{k=1}^K \alpha_k(x) \mathcal{N}(y|\mu_k(x), \Sigma_k(x))$$

where the network backbone outputs the mixture weights ($\alpha_k$), means ($\mu_k$), and covariances ($\Sigma_k$) simultaneously. To ensure training stability and avoid numerical underflow in low-probability regions, the authors utilize a modified formulation optimized via the LogSumExp trick.

Figure 1: Illustration of increasing UQ complexity. Point estimates provide no uncertainty (left), while arbitrary generative models are powerful but data/compute intensive (right). Many scientific problems are well approximated by simpler multimodal mixture distributions.

Datasets

The authors evaluate their approach across four simulated physical benchmarks exhibiting intrinsic multimodality:

  1. Sinusoidal Inverse Problem: A classic multi-valued inverse mapping benchmark where a single output maps to multiple inputs.
  2. Partially Observed Gravitational Attractor: A 2D dissipative dynamical system where particles converge asymptotically to one of three celestial attractors.
  3. ODE Saddle-Node Bifurcation: A canonical differential equation system where a hidden bifurcation parameter creates distinct convergent and divergent trajectory branches.
  4. Lorenz System (Lorenz-63): A classic chaotic 3D system characterized by a double-lobed attractor, evaluated under observational noise.

Key Insights and Ideas

  • The Topological "Bridge" Problem: Flow-based and diffusion models are mathematically constructed as diffeomorphisms (smooth, invertible maps). Because they try to continuously stretch a single Gaussian base distribution into disconnected target modes, they are topologically forced to leave artificial "bridges" of non-zero probability in physically forbidden zones. MDNs bypass this entirely by summing discrete kernels.
  • Global vs. Local Updates: In diffusion models, learning the relative weights of distant modes requires samples to traverse low-density barriers, leading to poor sample efficiency. MDNs optimize the negative log-likelihood globally; a single rare observation immediately corrects the global mixture weights ($\alpha_k$).
  • Interpretability as Order Parameters: The mixture weights $\alpha_k(x)$ predicted by MDNs act naturally as physical order parameters. They map out phase diagrams, identify stability boundaries, and classify system regimes cleanly without requiring expensive post-hoc Monte Carlo sampling.
  • $O(1)$ Inference Latency: While diffusion models require tens or hundreds of sequential network evaluations to solve an ODE/SDE for a single prediction, MDNs require just one forward pass, making them ideal for real-time control loops.

Key Results

  • Data Efficiency in Low-Data Regimes: In the sinusoidal inverse problem and the saddle-node bifurcation task, MDNs drastically outperformed CFM when training data was scarce ($N \le 100$). For instance, in the bifurcation task with 50 samples, MDN achieved a test Negative Log-Likelihood (NLL) of $85.6 \pm 6.5$, whereas CFM lagged far behind at $691.2 \pm 146.5$.
Figure 3: Comparison of MDN and CFM on the sinusoidal inverse problem. Top: test NLL versus training set size, with bands showing 1 Std. Dev. over 12 seeds. Bottom: averaged NLL surfaces.
  • Chaotic Rollouts: When paired with a recurrent architecture (RNN-MDN) on the chaotic Lorenz system, the explicit mixture model successfully tracked the double-lobed geometry over long horizons without drifting. It achieved a Maximum Mean Discrepancy (MMD) of $3.813 \times 10^{-4}$, outperforming standard MSE baselines by two orders of magnitude while using fewer parameters.

Limitations and Future Work

Despite their strengths, MDNs introduce a key hyperparameter: the number of mixture components ($K$). Choosing $K$ requires domain expertise or careful cross-validation. Furthermore, while MDNs excel in low-to-moderate dimensions (typical of many physics problems), scaling explicit parametric mixtures to massive, unstructured, high-dimensional spaces (like video or raw image generation) remains an open challenge.

Moving forward, the authors suggest integrating MDNs with more expressive backbones like neural operators (for PDEs) and Kolmogorov-Arnold Networks (KANs) to further enhance interpretability.

Summary updated

Abstract

Scientific machine learning (SciML) increasingly requires models that capture multimodal conditional uncertainty arising from ill-posed inverse problems, multistability, and chaotic dynamics. While recent work has favored highly expressive implicit generative models such as diffusion and flow-based methods, these approaches are often data-hungry, computationally costly, and misaligned with the structured solution spaces frequently found in scientific problems. We demonstrate that Mixture Density Networks (MDNs) provide a principled yet largely overlooked alternative for multimodal uncertainty quantification in SciML. As explicit parametric density estimators, MDNs impose an inductive bias tailored to low-dimensional, multimodal physics, enabling direct global allocation of probability mass across distinct solution branches. This structure delivers strong data efficiency, allowing reliable recovery of separated modes in regimes where scientific data is scarce. We formalize these insights through a unified probabilistic framework contrasting explicit and implicit distribution networks, and demonstrate empirically that MDNs achieve superior generalization, interpretability, and sample efficiency across a range of inverse, multistable, and chaotic scientific regression tasks.
BibTeX
@misc{guilhoto2026multimodal,
  author = {Leonardo Ferreira Guilhoto and Akshat Kaushal and Paris Perdikaris},
  title = {{Multimodal Scientific Learning Beyond Diffusions and Flows}},
  year = {2026},
  eprint = {2602.00960},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.00960},
}
Figure