Introduction
Generative modeling aims to learn complex data distributions by transforming simple base distributions. Normalizing Flows (NF) are a popular class of such models, providing exact density estimation and efficient sampling. However, standard NFs often act as "black boxes," where the internal transformations lack physical intuition, and designing invertible mappings with tractable Jacobians remains a significant challenge.
Neural Hamiltonian Flows (NHF) address the complexity issue by leveraging Hamiltonian dynamics—the mathematical framework governing classical mechanics. Because Hamiltonian systems are naturally volume-preserving and reversible, they provide an elegant foundation for normalizing flows. However, typical NHF implementations parameterize both kinetic and potential energy with neural networks, leading to a "double black box" where the learned physical quantities may not correspond to any meaningful physical reality.
Methods: Fixed-kinetic Neural Hamiltonian Flows (FK-NHF)
To enhance interpretability and reduce model complexity, this work introduces Fixed-kinetic Neural Hamiltonian Flows (FK-NHF). In a standard Hamiltonian system, the total energy $H(q, p)$ is the sum of kinetic energy $K(p)$ and potential energy $V(q)$.
While standard approaches (MLPK-NHF) learn both $K$ and $V$ as Multi-Layer Perceptrons (MLPs), FK-NHF fixes the kinetic energy to a classical quadratic form: $K(p) = \frac{1}{2} p^T M^{-1} p$, where $M$ is a mass matrix. By fixing $K$, the model is forced to encapsulate the entire complexity of the target distribution within the learned potential energy $V(q)$.
The transformation is implemented using a Leapfrog integrator, a numerical method that preserves the symplectic structure of the dynamics, ensuring perfect invertibility. This framework is applicable to both standard generative modeling and Bayesian inference, where it can transform a prior distribution into a posterior given observed data.
Key Insights and Ideas
Physics as a Regularizer
Fixing the kinetic energy serves as a powerful inductive bias. In standard NHF, the model can "hide" complex features in the artificial momenta ($p$) generated by the encoder. By constraining $K(p)$ to a simple quadratic form, the potential energy $V(q)$ must directly account for the structure of the data, acting as a natural regularizer that prevents unphysical solutions.
Enhanced Interpretability
The primary advantage of FK-NHF is that the learned potential energy $V(q)$ naturally aligns with the negative log-density of the target distribution. This allows researchers to visualize the "energy landscape" of the model; local minima (valleys) in $V(q)$ correspond directly to the modes of the data distribution.
Robustness and Efficiency
FK-NHF demonstrates remarkable robustness to hyperparameters such as the integration time ($T$) and the number of steps ($L$). Unlike MLPK-NHF, which can be sensitive to these settings, FK-NHF maintains high performance even with a small number of integration steps. Furthermore, while diffusion models often require hundreds of iterations, FK-NHF achieves high-quality results in as few as 10 Leapfrog steps.
Key Results
The model was evaluated on synthetic 2D mixtures, image datasets (MNIST, Fashion-MNIST), and cosmological inference tasks.
Synthetic 2D Mixture and Interpretability
In tests on a 9-mode Gaussian mixture, FK-NHF consistently produced potential energy maps where the local minima perfectly centered on the data modes. In contrast, the unconstrained MLPK-NHF often produced "flipped" or otherwise uninterpretable energy landscapes, failing to provide physical insight into the data structure.
Image Generation and Bayesian Inference
On MNIST and Fashion-MNIST, FK-NHF achieved competitive Bits Per Pixel (BPP) and KL divergence scores, matching the performance of more complex models like RealNVP while maintaining a simpler architecture. In cosmological parameter inference using Type Ia supernovae data, FK-NHF provided results competitive with Hamiltonian Monte Carlo (HMC), producing uncorrelated samples that allow for faster convergence of empirical averages.
Limitations and Future Work
While promising, FK-NHF exhibits a small residual bias in Bayesian inference tasks, which could potentially be mitigated through importance sampling. Additionally, the choice and adaptation of the mass matrix $M$ significantly impact performance; future research into automated mass matrix tuning could further enhance the model's expressivity. Finally, scaling this physics-driven approach to higher-dimensional data, such as high-resolution images, remains an important area for future exploration.
Summary updated