Abstract
Characterizing the entropy of a system is a crucial, and often computationally costly, step in understanding its thermodynamics. It plays a key role in the study of phase transitions, pattern formation, protein folding, and more. Current methods for entropy estimation suffer from a high computational cost, lack of generality, or inaccuracy and inability to treat complex, strongly interacting systems. In this paper, we present a method, termed machine-learning iterative calculation of entropy (MICE), for calculating the entropy by iteratively dividing the system into smaller subsystems and estimating the mutual information between each pair of halves. The estimation is performed with a recently proposed machine-learning algorithm which works with arbitrary network architectures that can be chosen to fit the structure and symmetries of the system at hand. We show that our method can calculate the entropy of various systems, both thermal and athermal, with state-of-the-art accuracy. Specifically, we study various classical spin systems and identify the jamming point of a bidisperse mixture of soft disks. Finally, we suggest that besides its role in estimating the entropy, the mutual information itself can provide an insightful diagnostic tool in the study of physical systems.
Introduction
In statistical mechanics, entropy ($S$) is the fundamental quantity for understanding phase transitions and material properties. However, calculating $S = -\sum p_i \log p_i$ directly is often impossible because the number of microstates grows exponentially with system size—a challenge known as the "curse of dimensionality." Traditional approaches, such as thermodynamic integration of specific heat, are computationally expensive and limited to thermal systems. Data compression algorithms, while faster, often fail to capture long-range correlations or the complex spatial geometries of physical systems.
The MICE Framework
We introduce Machine-learning Iterative Calculation of Entropy (MICE), a scalable method that leverages the power of neural networks to estimate entropy through a recursive "divide and conquer" strategy. MICE is based on the information-theoretic identity:
$$S(X) = S(X_1^1) + S(X_1^2) - \mathcal{M}(X_1^1, X_1^2)$$
where $\mathcal{M}$ is the mutual information between two halves of the system.
The process involves iteratively halving the system until reaching a minimal size where the entropy can be calculated exactly. The critical component is the Mutual Information Neural Estimation (MINE). Instead of estimating the full probability distribution, MINE uses artificial neural networks (ANNs) to solve an optimization problem that yields the mutual information. By using Convolutional Neural Networks (CNNs), MICE naturally respects the spatial symmetries of physical lattices, a significant advantage over standard 1D compression algorithms.
Benchmarking and Results
MICE was tested against several canonical models in condensed matter physics. In the 2D Ferromagnetic Ising model, MICE achieved state-of-the-art accuracy, with errors consistently below $0.06 k_B$ per spin. It successfully captured the phase transition and the complex ground states of the Anti-ferromagnetic Ising model on a triangular lattice, a system characterized by high frustration where other methods often struggle.
Identifying the Jamming Transition
A significant challenge in granular physics is identifying the "jamming point" ($\phi_J$) in athermal systems, such as bidisperse soft disks. At this point, the system transitions from a fluid-like state to a rigid, disordered solid. Previous attempts to locate this transition using compression-based entropy estimation failed to detect the subtle structural changes.
As shown in Figure 4, MICE successfully identifies the jamming point at a packing fraction of $\phi \approx 0.84$. The mutual information $\mathcal{M}$ serves as a sensitive diagnostic tool, exhibiting a clear signature at the transition that other algorithms miss.
Discussion and Conclusion
Beyond its utility in entropy calculation, the mutual information itself provides deep physical insights. It obeys an "area law" at high temperatures, where $\mathcal{M}$ scales with the boundary size between subsystems rather than the volume. MICE exploits this property to maintain computational efficiency, with costs scaling only logarithmically with system size.
The versatility of MICE makes it a promising tool for a wide range of applications, from calculating entanglement entropy in quantum systems to studying out-of-equilibrium active matter. By combining fundamental information theory with modern machine learning, MICE offers a robust path toward measuring the "unmeasurable" in complex physical systems.
Summary updated