Within-layer Diversity Reduces Generalization Gap

Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, Moncef GabboujZenodo 2021 · DOI · OpenAlex

Introduction

A fundamental challenge in deep learning theory is explaining why over-parameterized neural networks generalize well despite having the capacity to overfit training data. While empirical techniques like Dropout suggest that reducing redundancy is beneficial, a formal theoretical link between feature diversity and generalization has remained elusive. Recent research provides a rigorous framework proving that feature diversity within hidden layers directly reduces the generalization gap. By quantifying diversity as the average $L_2$-distance between neuron activations, this work derives new Rademacher complexity-based bounds that establish "distinctness" as a mathematical driver of model performance.

Theoretical Framework

The study introduces a formal diversity metric based on representation diversity—how distinct the outputs (activations) of neurons are for a given input—rather than just weight diversity.

The Diversity Metric

For a layer with $M$ neurons, diversity is defined by the average pairwise $L_2$-distance between activations across the dataset. Low distance indicates redundancy, while high distance indicates that neurons have learned distinct features.

Rademacher Complexity and Generalization

The core theoretical contribution involves bounding the Rademacher complexity of a network class using a minimum diversity factor ($d_{min}$). The researchers prove that the generalization bound is a decreasing function of this diversity. For a two-layer network, the generalization gap scales as $\sim (C - d^2_{min}) / \sqrt{N}$, where $C$ is a constant and $N$ is the number of samples. This implies that as diversity increases, the upper bound on the generalization error shrinks. This result is extended to deep multi-layer networks and various loss functions, including Hinge and Cross-Entropy.

Empirical Validation

The theoretical findings were validated across several benchmark datasets, including MNIST, CIFAR-10, and ImageNet.

Evolution of feature diversity across training epochs for different ResNet architectures.

On MNIST, the derived theoretical bounds tracked the actual test-train error gap with a correlation exceeding 0.99. Experiments on larger models like ResNet architectures (Figure 1) show that standard Stochastic Gradient Descent (SGD) naturally optimizes for diversity over time. Notably, deeper architectures (e.g., ResNet50) exhibit higher levels of feature diversity compared to shallower ones (e.g., ResNet18), suggesting that depth inherently facilitates the learning of distinct representations.

Discussion

The research highlights several key insights:

  • Implicit Bias of SGD: Diversity appears to be an implicit bias of SGD training, as models naturally migrate toward high-diversity regions without explicit regularization.
  • Redundancy Costs: Redundant neurons effectively increase the Rademacher complexity, leading to looser generalization bounds and less predictable performance on unseen data.
  • Classification Gains: For binary classification with logistic loss, the generalization bound decreases exponentially with respect to feature diversity, underscoring its importance in discriminative tasks.

Conclusion and Future Work

By formalizing the intuition that "distinct features help," this work provides a new lens for understanding neural network generalization. Future research directions include investigating the tightness of these bounds in broader contexts, exploring the "more parameters" paradox where increased width does not always yield higher diversity, and examining the role of diversity in adversarial robustness.

Summary updated

Abstract

Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional ’between-layer’ feedback with additional ’within-layer’ feedback to encourage diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer’s overall diversity. By penalizing similarities and promoting diversity, we encourage each neuron to learn a distinctive representation and, thus, to enrich the data representation learned within the layer and to increase the total capacity of the model. We theoretically and empirically study how the within-layer activation diversity affects the generalization performance of a neural network and prove that increasing the diversity of hidden activations reduces the generalization gap.
BibTeX
@inproceedings{laakom2021within,
  author = {Firas Laakom and Jenni Raitoharju and Alexandros Iosifidis and Moncef Gabbouj},
  title = {{Within-layer Diversity Reduces Generalization Gap}},
  booktitle = {Zenodo},
  year = {2021},
  doi = {10.5281/zenodo.5772760},
}

Loading related content...

Figure