Abstract
Probabilistic models can be defined by an energy function, where the probability of each state is proportional to the exponential of the state's negative energy. This paper considers a generalization of energy-based models in which the probability of a state is proportional to an arbitrary positive, strictly decreasing, and twice differentiable function of the state's energy. The precise shape of the nonlinear map from energies to unnormalized probabilities has to be learned from data together with the parameters of the energy function. As a case study we show that the above generalization of a fully visible Boltzmann machine yields an accurate model of neural activity of retinal ganglion cells. We attribute this success to the model's ability to easily capture distributions whose probabilities span a large dynamic range, a possible consequence of latent variables that globally couple the system. Similar features have recently been observed in many datasets, suggesting that our new method has wide applicability.
Introduction
Standard energy-based models (EBMs), such as the Boltzmann machine, typically assume a fixed exponential relationship between a state's energy and its probability ($p \propto e^{-E}$). While this framework is foundational in statistical mechanics and machine learning, it often struggles to capture the complex, higher-order interactions prevalent in biological and physical systems. To address these deficiencies, researchers often resort to adding higher-order energy terms—which leads to a combinatorial explosion of parameters—or employing hidden variables, which can be difficult to interpret and computationally expensive to train.
This paper introduces a semiparametric generalization of EBMs that replaces the rigid exponential mapping with a learned, strictly decreasing nonlinearity $V(E)$. By learning the "shape" of this mapping directly from data, the model can capture global, system-wide coupling with significantly fewer parameters than traditional latent-variable models like Restricted Boltzmann Machines (RBMs).
Methodology
The proposed model defines the probability of a state $\mathbf{s}$ as: $$p(\mathbf{s}; \alpha, V) = \frac{e^{-V(E(\mathbf{s}; \alpha))}}{Z(\alpha, V)}$$ where $E$ is a standard energy function (e.g., a pairwise Ising model) and $V$ is a learned nonlinearity.
To ensure $V$ remains strictly increasing and twice differentiable, the authors parameterize it using a second-order differential equation, $V'' = WV'$, where $W$ is a piecewise constant function. This allows for smooth, unconstrained optimization. A critical challenge in learning such models is estimating the partition function $Z$. The authors utilize the Wang-Landau algorithm, a Monte Carlo technique from statistical mechanics, to estimate the "density of states"—the number of configurations at each energy level. This approach enables the model to learn effectively even from states with extremely low probabilities.
Results and Insights
The model was evaluated using simultaneous recordings of 160 retinal ganglion cells from a salamander and patches of natural images.
Performance and Scaling
The semiparametric pairwise model significantly outperformed standard pairwise and "K-pairwise" models. Remarkably, it achieved performance comparable to an RBM with $N/2$ hidden units while using orders of magnitude fewer parameters. For instance, while an RBM might require thousands of parameters for its hidden layer, the semiparametric model achieved a similar fit by adding only about 12 parameters to define the nonlinearity $V$.
Global Coupling and Latent Variables
The nonlinearity $V$ acts as a "global coupler," defining how the entire population's energy relates to its probability rather than focusing on individual pairwise interactions. The authors demonstrate a mathematical equivalence between this semiparametric form and a system with a single fluctuating latent variable (e.g., a fluctuating "temperature"). Furthermore, in the limit of large systems, the optimal nonlinearity $V(E)$ appears to converge toward the microcanonical entropy, providing a deep theoretical link to statistical mechanics.
Conclusion
By learning the mapping between energy and probability, semiparametric EBMs offer a powerful and efficient alternative to traditional latent-variable models for capturing global dependencies. While estimating the density of states may become computationally intensive in extremely high-dimensional or "rugged" energy landscapes, the method's success in neural and image data suggests broad applicability. Future work includes integrating these global nonlinearities into deeper architectures like Deep Belief Networks and exploring their utility in other domains characterized by global coupling, such as protein folding or financial markets.
Summary updated