Understanding Deep Learning Generalization by Maximum Entropy

Guanhua Zheng, Jitao Sang, Changsheng XuarXiv 2017 · arXiv

Overview

This research provides a novel theoretical framework for deep learning generalization by framing Deep Neural Networks (DNNs) as a recursive implementation of the Maximum Entropy (ME) principle. The authors prove that DNN hidden layers act as a sequence of feature learners designed to satisfy specific entropy conditions, offering a unified explanation for why overparameterized models generalize well and why architectural choices like shortcuts and regularization are effective.

The Generalization Paradox

A central paradox in modern AI is why deep neural networks—which often have far more parameters than training samples—generalize so well to unseen data instead of simply memorizing the training set. Classical statistical learning theories suggest that such complex models should suffer from massive overfitting.

A toy example fitting 5 data points, illustrating the conceptual definitions of underfitting and overfitting through the lens of data hypothesis.

The authors tackle this "generalization puzzle" by looking at the Maximum Entropy principle, which states that the best model is the one that makes the fewest assumptions (has the highest entropy) while still satisfying the constraints of the observed data.

Theoretical Framework: The ME Equivalence Theorem

The paper introduces a framework where the multi-layer structure of a DNN works to transform raw data into features that allow the model to reach a Maximum Entropy state.

Conceptual comparison of model types: non-ME models, original ME models, and simple models incorporating feature constraints.

The authors derive two necessary and sufficient conditions for a feature-based softmax model to be equivalent to an original ME model:

  • Condition 1 (Equivalent Condition): The input $X$ and label $Y$ must be conditionally independent given the features $T$.
  • Condition 2 (Solvable Condition): All individual features within $T$ must be conditionally independent given the label $Y$.

Recursive Decomposition in DNNs

Because these conditions are extremely difficult to satisfy in a single step, the authors propose that DNNs solve this via recursive decomposition. They prove that a standard DNN (specifically those using sigmoid activations and a softmax output) can be viewed as a multi-stage process that breaks down the complex ME optimization problem into smaller, manageable sub-problems at each layer. Each layer progressively refines features to meet the Equivalence Theorem's conditions.

Key Insights and Unification

This theory provides a formal link between Maximum Entropy and several practical DNN design choices:

  • Information Bottleneck (IB): The ME objective is a sufficient condition for IB theory, explaining why DNNs naturally maximize label information $I(T;Y)$ while minimizing input information $I(X;T)$.
  • Shortcuts: In the context of ME, "shortcuts" (like those in ResNets) help preserve essential information from the original input during the recursive decomposition, preventing information loss in deep architectures.
  • Regularization: Techniques like $L2$ normalization and dropout help the model approximate the "solvable condition," ensuring features remain independent enough to satisfy the ME principle.

Conclusion

By framing DNNs as entropy maximizers, this work offers a mathematical foundation for understanding why deep architectures work. While currently focused on sigmoid-activated layers, it suggests a path toward designing new architectures based on information-theoretic principles rather than empirical trial and error.

Summary updated

Abstract

Deep learning achieves remarkable generalization capability with overwhelming number of model parameters. Theoretical understanding of deep learning generalization receives recent attention yet remains not fully explored. This paper attempts to provide an alternative understanding from the perspective of maximum entropy. We first derive two feature conditions that softmax regression strictly apply maximum entropy principle. DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle. The connection between DNN and maximum entropy well explains why typical designs such as shortcut and regularization improves model generalization, and provides instructions for future model development.
BibTeX
@misc{zheng2017understanding,
  author = {Guanhua Zheng and Jitao Sang and Changsheng Xu},
  title = {{Understanding Deep Learning Generalization by Maximum Entropy}},
  year = {2017},
  eprint = {1711.07758},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.07758},
}
Figure