Parsimonious neural networks learn classical mechanics and can teach it

Saaketh Desai, Alejandro StrachanarXiv 2020 · arXiv

Introduction

Machine learning (ML) has revolutionized pattern recognition across numerous domains. However, its application in the physical sciences is often hampered by two significant challenges: interpretability and generalizability. Standard "black box" neural networks, while capable of fitting complex datasets with high precision, frequently fail to respect fundamental physical constraints such as the conservation of energy or time-reversibility. In fields like materials science, where data collection is often expensive and scarce, models that lack an "understanding" of underlying physical laws are prone to failure when extrapolated to novel scenarios.

To bridge this gap, researchers have introduced Parsimonious Neural Networks (PNNs). This framework leverages the principle of Occam’s Razor—parsimony—to force neural networks toward the simplest possible mathematical explanations for observed data. By prioritizing simplicity, these networks do more than just predict outcomes; they rediscover the governing laws of physics.

Methods: Evolving Simplicity

The core innovation of the PNN lies in its training methodology. Unlike traditional networks that optimize weights solely via backpropagation, PNNs utilize evolutionary optimization through genetic algorithms to evolve the network's structure and parameters simultaneously.

Schematic of the Parsimonious Neural Network (PNN) architecture featuring a force sub-net.

The architecture is designed to be flexible, allowing the genetic algorithm to select from various activation functions (e.g., linear, ReLU, tanh, ELU) and weight types. A critical component is the "tax" on complexity integrated into the objective function. This penalty term ($p$) discourages unnecessary complexity by making non-linear activations "more expensive" than linear ones and favoring fixed, simple weights (such as 0, 1, or $\Delta t$) over trainable floating-point values.

For mechanics-related tasks, the researchers included a pre-trained "force sub-net" to provide the concept of acceleration, though the framework is capable of learning such concepts from scratch given sufficient data. By varying the parsimony coefficient, the system generates a "Pareto front"—a set of models representing the optimal trade-offs between predictive accuracy and mathematical simplicity.

Key Insights: Rediscovering Physical Laws

The PNN framework was tested on two distinct problems: particle dynamics in a Lennard-Jones potential and the prediction of material melting temperatures.

In the classical mechanics test, the PNN achieved a remarkable breakthrough. To maintain accuracy while minimizing complexity, the network "discovered" that it needed to evaluate forces at the midpoint between time steps. This specific configuration is the mathematical foundation of the Verlet integrator, a standard tool in physics simulations known for its energy conservation and time-reversibility. The PNN-derived equations matched the Position Verlet algorithm with an error in mass of only 3 parts in 10,000.

Similarly, in the study of melting points across 218 materials, the PNNs did not merely find statistical correlations. Instead, they converged on relationships closely resembling the Lindemann melting law, a physical principle established in 1910. Interestingly, the PNNs identified slightly more complex variations that outperformed the original Lindemann law on the Pareto front, suggesting that machine learning can refine and extend long-standing physical intuitions.

Results and Performance

The advantages of parsimony are most evident when comparing PNNs to standard Feed-Forward Neural Networks (FFNNs). While FFNNs can achieve low training error, they often suffer from "energy drift"—a gradual, unphysical change in total energy over time—and fail to accurately reverse trajectories in time.

Performance metrics comparing PNNs to standard FFNNs, highlighting error rates, energy conservation, and time-reversibility.

As shown in the comparative analysis, PNNs maintain excellent long-term energy conservation, matching the stability of state-of-the-art physics integrators. Furthermore, because PNNs favor linear activations and integer-like weights, the resulting models are inherently interpretable. A "trained" PNN can be directly translated into a standard algebraic equation, allowing scientists to inspect and verify the learned physics.

Conclusion and Future Directions

Parsimonious Neural Networks demonstrate that simplicity is not just an aesthetic preference but a functional requirement for robust, interpretable physical modeling. By "taxing" complexity, these networks can teach us the underlying rules of the systems they observe.

While the computational cost of genetic algorithms remains a challenge compared to backpropagation, the potential for discovery is vast. Future work aims to expand the search space to deeper architectures, potentially allowing for the discovery of higher-order physics or entirely new physical laws from raw observational data without prior knowledge of forces.

Summary updated

Abstract

Machine learning is playing an increasing role in the physical sciences and significant progress has been made towards embedding domain knowledge into models. Less explored is its use to discover interpretable physical laws from data. We propose parsimonious neural networks (PNNs) that combine neural networks with evolutionary optimization to find models that balance accuracy with parsimony. The power and versatility of the approach is demonstrated by developing models for classical mechanics and to predict the melting temperature of materials from fundamental properties. In the first example, the resulting PNNs are easily interpretable as Newton's second law, expressed as a non-trivial time integrator that exhibits time-reversibility and conserves energy, where the parsimony is critical to extract underlying symmetries from the data. In the second case, the PNNs not only find the celebrated Lindemann melting law, but also new relationships that outperform it in the pareto sense of parsimony vs. accuracy.
BibTeX
@misc{desai2020parsimonious,
  author = {Saaketh Desai and Alejandro Strachan},
  title = {{Parsimonious neural networks learn classical mechanics and can teach it}},
  year = {2020},
  eprint = {2005.11144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.11144},
}
Figure