Fixed-kinetic Neural Hamiltonian Flows for enhanced interpretability and reduced complexity

Vincent Souveton, Arnaud Guillin, Jens Jasche, Guilhem Lavaux, Manon MichelAISTATS 2024

Introduction

Generative modeling aims to learn complex data distributions by transforming simple base distributions. Normalizing Flows (NF) are a popular class of such models, providing exact density estimation and efficient sampling. However, standard NFs often act as "black boxes," where the internal transformations lack physical intuition, and designing invertible mappings with tractable Jacobians remains a significant challenge.

Neural Hamiltonian Flows (NHF) address the complexity issue by leveraging Hamiltonian dynamics—the mathematical framework governing classical mechanics. Because Hamiltonian systems are naturally volume-preserving and reversible, they provide an elegant foundation for normalizing flows. However, typical NHF implementations parameterize both kinetic and potential energy with neural networks, leading to a "double black box" where the learned physical quantities may not correspond to any meaningful physical reality.

Methods: Fixed-kinetic Neural Hamiltonian Flows (FK-NHF)

To enhance interpretability and reduce model complexity, this work introduces Fixed-kinetic Neural Hamiltonian Flows (FK-NHF). In a standard Hamiltonian system, the total energy $H(q, p)$ is the sum of kinetic energy $K(p)$ and potential energy $V(q)$.

While standard approaches (MLPK-NHF) learn both $K$ and $V$ as Multi-Layer Perceptrons (MLPs), FK-NHF fixes the kinetic energy to a classical quadratic form: $K(p) = \frac{1}{2} p^T M^{-1} p$, where $M$ is a mass matrix. By fixing $K$, the model is forced to encapsulate the entire complexity of the target distribution within the learned potential energy $V(q)$.

Schematic of the Neural Hamiltonian Flow architecture, showing the reversible transformation between the prior $\pi_0$ and target $\pi$ distributions via Leapfrog (LF) integration steps.

The transformation is implemented using a Leapfrog integrator, a numerical method that preserves the symplectic structure of the dynamics, ensuring perfect invertibility. This framework is applicable to both standard generative modeling and Bayesian inference, where it can transform a prior distribution into a posterior given observed data.

Key Insights and Ideas

Physics as a Regularizer

Fixing the kinetic energy serves as a powerful inductive bias. In standard NHF, the model can "hide" complex features in the artificial momenta ($p$) generated by the encoder. By constraining $K(p)$ to a simple quadratic form, the potential energy $V(q)$ must directly account for the structure of the data, acting as a natural regularizer that prevents unphysical solutions.

Enhanced Interpretability

The primary advantage of FK-NHF is that the learned potential energy $V(q)$ naturally aligns with the negative log-density of the target distribution. This allows researchers to visualize the "energy landscape" of the model; local minima (valleys) in $V(q)$ correspond directly to the modes of the data distribution.

Robustness and Efficiency

FK-NHF demonstrates remarkable robustness to hyperparameters such as the integration time ($T$) and the number of steps ($L$). Unlike MLPK-NHF, which can be sensitive to these settings, FK-NHF maintains high performance even with a small number of integration steps. Furthermore, while diffusion models often require hundreds of iterations, FK-NHF achieves high-quality results in as few as 10 Leapfrog steps.

Key Results

The model was evaluated on synthetic 2D mixtures, image datasets (MNIST, Fashion-MNIST), and cosmological inference tasks.

Synthetic 2D Mixture and Interpretability

In tests on a 9-mode Gaussian mixture, FK-NHF consistently produced potential energy maps where the local minima perfectly centered on the data modes. In contrast, the unconstrained MLPK-NHF often produced "flipped" or otherwise uninterpretable energy landscapes, failing to provide physical insight into the data structure.

Comparison of learned potential energy landscapes for MLPK-NHF (top) and FK-NHF (bottom) across different prior distributions. FK-NHF landscapes clearly show modes corresponding to the data distribution.

Image Generation and Bayesian Inference

On MNIST and Fashion-MNIST, FK-NHF achieved competitive Bits Per Pixel (BPP) and KL divergence scores, matching the performance of more complex models like RealNVP while maintaining a simpler architecture. In cosmological parameter inference using Type Ia supernovae data, FK-NHF provided results competitive with Hamiltonian Monte Carlo (HMC), producing uncorrelated samples that allow for faster convergence of empirical averages.

Limitations and Future Work

While promising, FK-NHF exhibits a small residual bias in Bayesian inference tasks, which could potentially be mitigated through importance sampling. Additionally, the choice and adaptation of the mass matrix $M$ significantly impact performance; future research into automated mass matrix tuning could further enhance the model's expressivity. Finally, scaling this physics-driven approach to higher-dimensional data, such as high-resolution images, remains an important area for future exploration.

Summary updated

BibTeX
@inproceedings{souveton2024fixed,
  author = {Vincent Souveton and Arnaud Guillin and Jens Jasche and Guilhem Lavaux and Manon Michel},
  title = {{Fixed-kinetic Neural Hamiltonian Flows for enhanced interpretability and reduced complexity}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
}
Figure