Structure Preserving Neural Networks: A Case Study in the Entropy Closure of the Boltzmann Equation

Steffen Schotthöfer, Tianbai Xiao, Martin Frank, Cory D. HauckICML 2022 · OpenReview

Introduction

The Boltzmann equation is a cornerstone of statistical physics, describing how particles move and collide in a gas. However, solving it directly is computationally prohibitive because it operates in a high-dimensional "phase space" encompassing both 3D position and 3D velocity.

To make these simulations tractable, researchers rely on moment methods, which reduce dimensionality by tracking macroscopic averages (moments) like density, momentum, and energy. This introduces the "closure problem": equations for lower-order moments inevitably depend on higher-order ones that aren't being tracked.

The gold standard for solving this is the $M_N$ closure, which uses an entropy minimization principle to reconstruct the particle distribution. While physically accurate and stable, it requires solving a complex optimization problem in every single grid cell at every time step. In high-performance simulations, this single step can consume over 80% of the total computation time.

Recent research introduces specialized neural network architectures that act as ultra-fast surrogates for this expensive "entropy closure" step. By enforcing mathematical constraints directly into the network design, these models achieve up to a 1000x speedup while strictly preserving essential physical properties.

Structure-Preserving Methods

Rather than using standard "black box" neural networks, which can hallucinate unphysical results like negative entropy production, the authors employ structure-preserving architectures to ensure physical validity:

  1. Input Convex Neural Networks (ICNNs): These networks are designed so that their output is always a convex function of the input. By learning the mapping from moments to entropy, the network's gradients automatically provide the optimal Lagrange multipliers needed for the closure.
  2. Monotonic Neural Networks: An alternative approach maps moments directly to Lagrange multipliers, utilizing a penalty function during training to enforce required monotonicity.
  3. Sobolev Training: The networks are trained to match not just values, but also derivatives (gradients). This significantly improves accuracy and generalization.
The workflow of the Input Convex Neural Network closure, showing how moments are normalized and processed to predict entropy and Lagrange multipliers.

Data Generation and Key Insights

Training these models requires careful data generation. The "realizable set"—the space of valid moments—has a complex boundary where the underlying mathematics becomes ill-conditioned.

Instead of uniform sampling, the researchers used a low-discrepancy sequence combined with an "eigenvalue bound." By using the smallest eigenvalue of the Hessian as a proxy for distance to the boundary, they could systematically sample the most difficult regions of the moment space where traditional solvers struggle.

A key insight of this work is leveraging the duality between entropy and Lagrange multipliers. Training on the entropy functional and using its automatic differentiation to obtain multipliers is far more robust than attempting to learn the multipliers directly.

Results

The structure-preserving neural network closures demonstrated impressive performance on synthetic 1D and 2D moment systems:

  • Speed: The neural network surrogate is significantly faster than classical Newton-based optimizers, achieving speedups on the order of $O(10^3)$ for a single closure evaluation.
  • Accuracy: The neural closures captured system dynamics with relative errors typically between 1% and 5%.
  • Stability: Unlike standard neural networks, these models successfully maintained the mathematical entropy dissipation of the system over time, perfectly matching the behavior of expensive benchmark solvers.

Limitations and Future Work

The primary limitation of this approach is maintaining accuracy near the boundary of the realizable set. When moments approach the edge of what is physically possible (such as in vacuum states or extreme gradients), the entropy and multipliers grow rapidly toward infinity. This asymptotic behavior makes it incredibly difficult for any neural network to fit perfectly.

Behavior of Lagrange multipliers and entropy near the boundaries of the realizable set, illustrating the steep gradients that challenge neural network fitting.

Future work will need to focus on improving data generation and network performance specifically in these boundary regions. Additionally, scaling this approach to higher-order moment systems and more complex 3D geometries, potentially through adaptive sampling during training, remains an exciting frontier for machine learning in statistical physics.

Summary updated

Abstract

In this paper, we explore applications of deep learning in statistical physics. We choose the Boltzmann equation as a typical example, where neural networks serve as a closure to its moment system....
BibTeX
@inproceedings{schotthofer2022structure,
  author = {Steffen Schotthöfer and Tianbai Xiao and Martin Frank and Cory D. Hauck},
  title = {{Structure Preserving Neural Networks: A Case Study in the Entropy Closure of the Boltzmann Equation}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=u4H7PHy2ASS},
}
Figure