Maximum Multiscale Entropy and Neural Network Regularization

Amir R. Asadi, Emmanuel AbbearXiv 2020 · OpenReview

Abstract

This paper generalizes the classical principle of maximum entropy to a "multiscale" setting, where uncertainty is maximized across multiple levels of resolution simultaneously. By drawing a formal link between information theory and the Renormalization Group in physics, the authors derive a new class of regularizers that provide tighter generalization bounds for neural networks than standard single-scale methods.

Introduction

In statistical mechanics and machine learning, a fundamental result states that the distribution maximizing entropy under a mean constraint is the Gibbs-Boltzmann distribution. This principle is the bedrock of many regularization techniques. However, classical entropy is "single-scale" — it treats a system as a single block of variables.

Real-world systems, especially deep neural networks, are inherently multiscale. A neural network processes information through a hierarchy of layers, where each layer represents a different "scale" of the hypothesis space. The authors argue that standard single-scale entropy regularizers fail to exploit this compositional structure. They set out to answer: How do we define and maximize entropy across multiple scales, and can this help us understand why deep networks generalize so well?

Methods

The authors introduce Multiscale Entropy, defined as a linear mixture of entropies calculated at different length scales. To find the distribution that maximizes this new objective, they propose a three-phase algorithmic framework:

  1. Initialization: Start with a "microscopic" Gibbs distribution at the finest scale.
  2. Renormalization (Coarse-graining): Iteratively eliminate degrees of freedom to move from fine to coarse scales. This involves "tilting" or scaling the distributions, a process the authors show is mathematically analogous to the Renormalization Group (RG) procedure used in statistical physics to study phase transitions.
  3. Refinement: Reconstruct the full distribution by concatenating these intermediate "scale-specific" distributions using conditional probabilities.
Algorithm 1: The multiscale entropy maximization procedure involving coarse-graining and renormalization steps.

Crucially, they prove that for a specific type of scale transformation called decimation (which maps directly to the layer-by-layer structure of residual networks), if the initial distribution is Gaussian, the resulting multiscale distribution remains a multivariate Gaussian.

Theoretical Insights

  • The Physics Connection: The most striking idea is the formal mapping between neural network training and the Renormalization Group. Maximizing multiscale entropy is essentially performing a Bayesian version of RG, where each layer of a network acts as a step in a coarse-graining process.
  • Data Processing Gain (DPG): The authors introduce DPG to quantify how much information is "filtered" or refined as you move across scales. They prove that the multiscale approach is always guaranteed to provide a tighter bound on excess risk than the single-scale approach because it accounts for this gain.
  • Layer-wise Complexity: Instead of controlling the complexity of the entire network at once, multiscale entropy allows for controlling the added complexity of each layer relative to the previous ones. This explains how deep networks can have millions of parameters but still generalize well—they aren't "using" all those degrees of freedom at every scale.

Key Results

  • Tighter Bounds: Theorem 5 proves that the multiscale Gibbs posterior achieves a strictly smaller excess risk bound compared to the classical single-scale Gibbs posterior.
  • Optimal Regularization: In their experiments, the authors show that there is an "optimal alpha" (a parameter controlling the scale-tilting). At this optimal point, the population risk is significantly lower than both standard Gibbs training and "random feature training" (where only the last layer is trained).
  • Gaussian Closure: They demonstrate that the MT (Marginalize-Tilt) algorithm is closed under the family of multivariate Gaussians, making the multiscale approach computationally feasible via parameter updates (mean and covariance) rather than complex density estimations.

Limitations and Future Work

  • Architecture Specificity: Much of the specific "decimation" results are tailored to residual networks (ResNets). While the general theory applies to any scale transformation, the "aha" moments regarding Gaussian distributions might not hold as cleanly for non-residual architectures or different activation functions.
  • Computational Overhead: While the paper provides an algorithm (MT), implementing this for billion-parameter models in a production environment would require more efficient approximations of the Hessian.
  • Beyond Decimation: The authors mention wavelet transformations as another scale-space example but do not explore them in the context of neural network regularization in this specific paper. Exploring other "scales" (like spatial resolution in CNNs) remains an open path.

Summary updated

Abstract

A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for instance in density estimation or to achieve excess risk bounds derived from single-scale entropy regularizers (Xu-Raginsky '17). This paper investigates a generalization of these results to a multiscale setting. We present different ways of generalizing the maximum entropy result by incorporating the notion of scale. For different entropies and arbitrary scale transformations, it is shown that the distribution maximizing a multiscale entropy is characterized by a procedure which has an analogy to the renormalization group procedure in statistical physics. For the case of decimation transformation, it is further shown that this distribution is Gaussian whenever the optimal single-scale distribution is Gaussian. This is then applied to neural networks, and it is shown that in a teacher-student scenario, the multiscale Gibbs posterior can achieve a smaller excess risk than the single-scale Gibbs posterior.
BibTeX
@misc{asadi2020maximum,
  author = {Amir R. Asadi and Emmanuel Abbe},
  title = {{Maximum Multiscale Entropy and Neural Network Regularization}},
  year = {2020},
  eprint = {2006.14614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.14614},
}
Figure