TL;DR
This paper presents a unified theoretical framework, rooted in statistical physics, to analyze deep neural networks (DNNs) and recurrent neural networks (RNNs). It reveals that despite structural differences, they behave identically in the infinite-width limit, but their finite-width behaviors and correction terms diverge, offering a path to understand their fundamental differences.
What Problem Does This Paper Address?
Deep learning has achieved remarkable success, but a deep theoretical understanding of why and how different network architectures work remains a challenge. While Bayesian inference on Gaussian processes has provided insights into the behavior of DNNs and RNNs in the infinite-width limit, a unified theoretical framework that can also account for finite-width effects and highlight architectural differences has been missing. Specifically, the relationship between the Gaussian process kernels of RNNs and DNNs, and how their finite-width corrections differ, has not been fully explored. This paper aims to bridge this gap by providing a systematic, first-principles derivation of the mean-field theory for both architectures, enabling a direct comparison of their capabilities and limitations.
Methods
The authors employ a field-theoretic approach, drawing from statistical physics of disordered systems, to derive a unified mean-field theory for both DNNs and RNNs. This involves:
- Bayesian Inference Framework: The analysis is framed within Bayesian supervised learning, where network priors are derived from parameter priors.
- Statistical Field Theory: Established techniques from statistical physics, particularly those used for disordered systems (like spin glasses), are adapted to model neural networks. This involves defining an "action" that encapsulates the network's dynamics.
- Saddle-Point Approximation: For large network widths (approaching infinity), a saddle-point approximation is used to simplify the complex integrals involved in calculating the network prior. This approximation becomes exact in the infinite-width limit.
- Auxiliary Variables: New auxiliary variables are introduced to handle the non-quadratic nature of activation functions and to decouple the system for analysis.
- Next-to-Leading-Order Corrections: Beyond the leading-order mean-field theory, the authors compute corrections of order $1/n$ (where $n$ is the network width) to account for finite-size effects. This involves analyzing the Hessian of the action.
The core novelty lies in applying these established statistical physics tools in a unified manner to both DNNs and RNNs, allowing for a direct comparison of their theoretical descriptions.
Datasets
Not discussed in the paper. The analysis is theoretical and relies on simulations to validate the derived mean-field predictions.
Key Insights and Ideas
- Unified Mean-Field Theory: A single theoretical framework can describe both DNNs and RNNs, unifying their analysis.
- Identical Infinite-Width Behavior: In the limit of infinite layer width (or time steps for RNNs), both architectures yield the same Gaussian process kernel when readouts are taken at a single point (layer or time step). This implies identical computational capabilities in this limit.
- Divergent Finite-Width Behavior: While identical in the infinite-width limit, the mean-field equations and their corrections differ significantly for finite widths. This is where the fundamental architectural differences emerge.
- Architecture-Specific Corrections: The next-to-leading-order corrections to the mean-field theory are architecture-specific, revealing how fluctuations propagate differently in DNNs (layer-wise) versus RNNs (temporally correlated due to weight sharing).
- RNNs' Slower Convergence: Numerically, RNNs tend to converge more slowly to the mean-field theory compared to DNNs, especially for longer sequences or deeper networks. This is attributed to the temporal correlations introduced by weight sharing.
Key Results
- Identical Gaussian Kernels: For infinite width, both DNNs and RNNs produce the same Gaussian process kernel for single-point readouts.
- Architecture-Specific Fluctuations: The $1/n$ corrections reveal that DNNs have layer-independent fluctuations, while RNNs exhibit temporally correlated fluctuations due to weight sharing.
- Numerical Convergence: Simulations show that RNNs generally converge slower to the mean-field predictions than DNNs, with convergence speed depending on prior parameters and network depth/time steps.
- Instability Threshold: For ReLU activations, the theory predicts an instability threshold ($g^2 > 2$) in RNNs where activity can diverge, a phenomenon less pronounced or absent in DNNs under similar conditions.
Limitations and Future Work
- Infinite-Width Limit: The core mean-field theory is derived in the infinite-width limit, and while corrections are computed, the analysis of finite-width effects is still an active area.
- Activation Functions: While the formalism is general, specific results and figures often focus on ReLU and erf activation functions. Extending to a broader range of nonlinearities could be explored.
- Computational Cost: The numerical simulations, especially for convergence studies, can be computationally intensive.
- Future Work: The authors suggest extending the formalism to investigate the fundamental differences between recurrent and deep architectures at finite widths. They also point to connecting their work with large-deviation theory and exploring the implications for understanding the expressivity of neural networks. The derived framework can also be used to compute finite-size corrections for the predictive distribution, building upon existing work.
Summary updated