Statistical physics, Bayesian inference and neural information processing

Erin Grant, Sandra Nestler, Berfin Şimşek, Sara A. Solla

Abstract

This paper explores the intersection of statistical physics, machine learning, and neuroscience to elucidate how neural networks—both artificial and biological—process information. We discuss a mathematical framework for network learning as a thermodynamic process, the application of Generalized Linear Models (GLMs) to biological neural spiking, and techniques for uncovering low-dimensional manifolds in high-dimensional neural data.

Introduction

Understanding the complex dynamics of neural systems is a significant mathematical challenge. In artificial intelligence, formalizing how a network generalizes from noisy data remains a core problem. In neuroscience, the advent of high-density recordings presents researchers with massive, noisy, high-dimensional datasets. This work addresses these challenges by bridging statistical physics with information processing, providing tools to describe learning ensembles, model time-dependent spiking, and extract meaningful low-dimensional representations of neural activity.

Thermodynamics of Learning and Bayesian Inference

Learning can be conceptualized as the systematic reduction of a network's configuration space. By framing the learning error as "energy" and the algorithm's tolerance to noise as "temperature," we can apply the principles of statistical mechanics to understand generalization.

Mapping from input space to output space, illustrating the distance $d$ between the model prediction $f_W(x)$ and the target $y$.

Using Gibbs distributions and Bayesian inference, we can calculate learning and generalization errors. A key insight from this framework is that overfitting can be viewed as a phase transition. This occurs when the "effective temperature" of the learning process—representing its tolerance for error—drops below the inherent noise level of the training data.

The error surface as a function of network weights, where learning is framed as finding the minimum "energy" state.

Modeling Biological Neurons with Generalized Linear Models (GLMs)

To accurately model biological neurons, neural spiking is treated as a Poisson process. GLMs predict a neuron's firing rate based on a linear combination of intrinsic covariates (past spiking history) and extrinsic covariates (sensory or motor inputs), passed through a nonlinear link function.

Unlike deep learning models trained via backpropagation through time, GLMs possess a log-convex likelihood. This ensures that gradient ascent will converge to a global optimum. Furthermore, the resulting model weights are highly interpretable, reflecting biological properties such as the refractory period. Incorporating intrinsic spiking history significantly improves model accuracy, as verified by the time-rescaling theorem.

Dimensionality Reduction and Neural Manifolds

While researchers may record from hundreds of neurons, the underlying dynamics of a specific task often reside on a much simpler, low-dimensional geometric structure known as a manifold. We distinguish between linear methods like Principal Component Analysis (PCA) and nonlinear methods like Isomap.

Linear methods like PCA identify the flat embedding space of neural data. However, nonlinear methods are required to "unroll" the true intrinsic manifold. For instance, in motor cortex studies, raw high-dimensional data may appear chaotic, but dimensionality reduction reveals highly organized clusters corresponding to specific motor intents, such as reaching toward different spatial targets.

Conclusion

The application of statistical physics to neural information processing provides a rigorous foundation for understanding learning and neural dynamics. While GLMs and manifold discovery techniques have limitations—such as assumptions of Poisson statistics or requirements for dense data sampling—they offer powerful, interpretable frameworks for decoding the brain's complex activity and understanding the fundamental limits of learning in artificial systems.

Summary updated

Abstract

Lecture notes from the course given by Professor Sara A. Solla at the Les Houches summer school on "Statistical physics of Machine Learning". The notes discuss neural information processing through the lens of Statistical Physics. Contents include Bayesian inference and its connection to a Gibbs description of learning and generalization, Generalized Linear Models as a controlled alternative to backpropagation through time, and linear and non-linear techniques for dimensionality reduction.
BibTeX
@article{grant2024statistical,
  author = {Erin Grant and Sandra Nestler and Berfin Şimşek and Sara A. Solla},
  title = {{Statistical physics, Bayesian inference and neural information processing}},
  journal = {Journal of Statistical Mechanics Theory and Experiment},
  year = {2024},
  doi = {10.1088/1742-5468/ad3350},
}
Figure