Unified field theoretical approach to deep and recurrent neuronal networks

Kai Segadlo, Bastian Epping, Alexander van Meegen, David Dahmen, M. Krämer, Moritz HeliasJournal of Statistical Mechanics Theory and Experiment 2022 · DOI

TL;DR

This paper presents a unified theoretical framework, rooted in statistical physics, to analyze deep neural networks (DNNs) and recurrent neural networks (RNNs). It reveals that despite structural differences, they behave identically in the infinite-width limit, but their finite-width behaviors and correction terms diverge, offering a path to understand their fundamental differences.

What Problem Does This Paper Address?

Deep learning has achieved remarkable success, but a deep theoretical understanding of why and how different network architectures work remains a challenge. While Bayesian inference on Gaussian processes has provided insights into the behavior of DNNs and RNNs in the infinite-width limit, a unified theoretical framework that can also account for finite-width effects and highlight architectural differences has been missing. Specifically, the relationship between the Gaussian process kernels of RNNs and DNNs, and how their finite-width corrections differ, has not been fully explored. This paper aims to bridge this gap by providing a systematic, first-principles derivation of the mean-field theory for both architectures, enabling a direct comparison of their capabilities and limitations.

Methods

The authors employ a field-theoretic approach, drawing from statistical physics of disordered systems, to derive a unified mean-field theory for both DNNs and RNNs. This involves:

  • Bayesian Inference Framework: The analysis is framed within Bayesian supervised learning, where network priors are derived from parameter priors.
  • Statistical Field Theory: Established techniques from statistical physics, particularly those used for disordered systems (like spin glasses), are adapted to model neural networks. This involves defining an "action" that encapsulates the network's dynamics.
  • Saddle-Point Approximation: For large network widths (approaching infinity), a saddle-point approximation is used to simplify the complex integrals involved in calculating the network prior. This approximation becomes exact in the infinite-width limit.
  • Auxiliary Variables: New auxiliary variables are introduced to handle the non-quadratic nature of activation functions and to decouple the system for analysis.
  • Next-to-Leading-Order Corrections: Beyond the leading-order mean-field theory, the authors compute corrections of order $1/n$ (where $n$ is the network width) to account for finite-size effects. This involves analyzing the Hessian of the action.

The core novelty lies in applying these established statistical physics tools in a unified manner to both DNNs and RNNs, allowing for a direct comparison of their theoretical descriptions.

Datasets

Not discussed in the paper. The analysis is theoretical and relies on simulations to validate the derived mean-field predictions.

Key Insights and Ideas

  • Unified Mean-Field Theory: A single theoretical framework can describe both DNNs and RNNs, unifying their analysis.
  • Identical Infinite-Width Behavior: In the limit of infinite layer width (or time steps for RNNs), both architectures yield the same Gaussian process kernel when readouts are taken at a single point (layer or time step). This implies identical computational capabilities in this limit.
  • Divergent Finite-Width Behavior: While identical in the infinite-width limit, the mean-field equations and their corrections differ significantly for finite widths. This is where the fundamental architectural differences emerge.
  • Architecture-Specific Corrections: The next-to-leading-order corrections to the mean-field theory are architecture-specific, revealing how fluctuations propagate differently in DNNs (layer-wise) versus RNNs (temporally correlated due to weight sharing).
  • RNNs' Slower Convergence: Numerically, RNNs tend to converge more slowly to the mean-field theory compared to DNNs, especially for longer sequences or deeper networks. This is attributed to the temporal correlations introduced by weight sharing.

Key Results

  • Identical Gaussian Kernels: For infinite width, both DNNs and RNNs produce the same Gaussian process kernel for single-point readouts.
  • Architecture-Specific Fluctuations: The $1/n$ corrections reveal that DNNs have layer-independent fluctuations, while RNNs exhibit temporally correlated fluctuations due to weight sharing.
  • Numerical Convergence: Simulations show that RNNs generally converge slower to the mean-field predictions than DNNs, with convergence speed depending on prior parameters and network depth/time steps.
  • Instability Threshold: For ReLU activations, the theory predicts an instability threshold ($g^2 > 2$) in RNNs where activity can diverge, a phenomenon less pronounced or absent in DNNs under similar conditions.

Limitations and Future Work

  • Infinite-Width Limit: The core mean-field theory is derived in the infinite-width limit, and while corrections are computed, the analysis of finite-width effects is still an active area.
  • Activation Functions: While the formalism is general, specific results and figures often focus on ReLU and erf activation functions. Extending to a broader range of nonlinearities could be explored.
  • Computational Cost: The numerical simulations, especially for convergence studies, can be computationally intensive.
  • Future Work: The authors suggest extending the formalism to investigate the fundamental differences between recurrent and deep architectures at finite widths. They also point to connecting their work with large-deviation theory and exploring the implications for understanding the expressivity of neural networks. The derived framework can also be used to compute finite-size corrections for the predictive distribution, building upon existing work.

Summary updated

Abstract

Abstract Understanding capabilities and limitations of different network architectures is of fundamental importance to machine learning. Bayesian inference on Gaussian processes has proven to be a viable approach for studying recurrent and deep networks in the limit of infinite layer width, n → ∞. Here we present a unified and systematic derivation of the mean-field theory for both architectures that starts from first principles by employing established methods from statistical physics of disordered systems. The theory elucidates that while the mean-field equations are different with regard to their temporal structure, they yet yield identical Gaussian kernels when readouts are taken at a single time point or layer, respectively. Bayesian inference applied to classification then predicts identical performance and capabilities for the two architectures. Numerically, we find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks and the convergence speed depends non-trivially on the parameters of the weight prior as well as the depth or number of time steps, respectively. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in 1/ n and we compute next-to-leading-order corrections which turn out to be architecture-specific. The formalism thus paves the way to investigate the fundamental differences between recurrent and deep architectures at finite widths n .
BibTeX
@article{segadlo2022unified,
  author = {Kai Segadlo and Bastian Epping and Alexander van Meegen and David Dahmen and M. Krämer and Moritz Helias},
  title = {{Unified field theoretical approach to deep and recurrent neuronal networks}},
  journal = {Journal of Statistical Mechanics Theory and Experiment},
  year = {2022},
  doi = {10.1088/1742-5468/ac8e57},
}
Figure