Abstract
This paper generalizes the classical principle of maximum entropy to a "multiscale" setting, where uncertainty is maximized across multiple levels of resolution simultaneously. By drawing a formal link between information theory and the Renormalization Group in physics, the authors derive a new class of regularizers that provide tighter generalization bounds for neural networks than standard single-scale methods.
Introduction
In statistical mechanics and machine learning, a fundamental result states that the distribution maximizing entropy under a mean constraint is the Gibbs-Boltzmann distribution. This principle is the bedrock of many regularization techniques. However, classical entropy is "single-scale" — it treats a system as a single block of variables.
Real-world systems, especially deep neural networks, are inherently multiscale. A neural network processes information through a hierarchy of layers, where each layer represents a different "scale" of the hypothesis space. The authors argue that standard single-scale entropy regularizers fail to exploit this compositional structure. They set out to answer: How do we define and maximize entropy across multiple scales, and can this help us understand why deep networks generalize so well?
Methods
The authors introduce Multiscale Entropy, defined as a linear mixture of entropies calculated at different length scales. To find the distribution that maximizes this new objective, they propose a three-phase algorithmic framework:
- Initialization: Start with a "microscopic" Gibbs distribution at the finest scale.
- Renormalization (Coarse-graining): Iteratively eliminate degrees of freedom to move from fine to coarse scales. This involves "tilting" or scaling the distributions, a process the authors show is mathematically analogous to the Renormalization Group (RG) procedure used in statistical physics to study phase transitions.
- Refinement: Reconstruct the full distribution by concatenating these intermediate "scale-specific" distributions using conditional probabilities.
Crucially, they prove that for a specific type of scale transformation called decimation (which maps directly to the layer-by-layer structure of residual networks), if the initial distribution is Gaussian, the resulting multiscale distribution remains a multivariate Gaussian.
Theoretical Insights
- The Physics Connection: The most striking idea is the formal mapping between neural network training and the Renormalization Group. Maximizing multiscale entropy is essentially performing a Bayesian version of RG, where each layer of a network acts as a step in a coarse-graining process.
- Data Processing Gain (DPG): The authors introduce DPG to quantify how much information is "filtered" or refined as you move across scales. They prove that the multiscale approach is always guaranteed to provide a tighter bound on excess risk than the single-scale approach because it accounts for this gain.
- Layer-wise Complexity: Instead of controlling the complexity of the entire network at once, multiscale entropy allows for controlling the added complexity of each layer relative to the previous ones. This explains how deep networks can have millions of parameters but still generalize well—they aren't "using" all those degrees of freedom at every scale.
Key Results
- Tighter Bounds: Theorem 5 proves that the multiscale Gibbs posterior achieves a strictly smaller excess risk bound compared to the classical single-scale Gibbs posterior.
- Optimal Regularization: In their experiments, the authors show that there is an "optimal alpha" (a parameter controlling the scale-tilting). At this optimal point, the population risk is significantly lower than both standard Gibbs training and "random feature training" (where only the last layer is trained).
- Gaussian Closure: They demonstrate that the MT (Marginalize-Tilt) algorithm is closed under the family of multivariate Gaussians, making the multiscale approach computationally feasible via parameter updates (mean and covariance) rather than complex density estimations.
Limitations and Future Work
- Architecture Specificity: Much of the specific "decimation" results are tailored to residual networks (ResNets). While the general theory applies to any scale transformation, the "aha" moments regarding Gaussian distributions might not hold as cleanly for non-residual architectures or different activation functions.
- Computational Overhead: While the paper provides an algorithm (MT), implementing this for billion-parameter models in a production environment would require more efficient approximations of the Hessian.
- Beyond Decimation: The authors mention wavelet transformations as another scale-space example but do not explore them in the context of neural network regularization in this specific paper. Exploring other "scales" (like spatial resolution in CNNs) remains an open path.
Summary updated