Margin Preservation of Deep Neural Networks

Jure Sokolic, Raja Giryes, Guillermo Sapiro, Miguel R. D. Rodrigues

Abstract

The generalization error of deep neural networks (DNNs) is traditionally expected to scale with model complexity. However, empirical evidence suggests that deep and wide networks often generalize remarkably well. This work investigates this phenomenon through the lens of the classification margin and the network's Jacobian matrix. We demonstrate that a bounded spectral norm of the Jacobian in the vicinity of training samples is a sufficient condition for generalization, regardless of network depth or width. This theoretical insight leads to a novel Jacobian-based regularizer that improves performance across various architectures and datasets, including MNIST, CIFAR-10, and ImageNet.

Introduction

A central challenge in deep learning theory is explaining why over-parameterized models do not suffer from massive overfitting. Classical measures like VC-dimension suggest that as the number of parameters grows, the generalization gap should widen. In practice, the opposite is often observed. This paper addresses this "generalization paradox" by shifting the focus from parameter counts to the geometry of the decision boundary, specifically the classification margin.

Theoretical Framework: The Jacobian and Robustness

The authors utilize the framework of Algorithmic Robustness, which posits that a model generalizes well if similar inputs yield similar loss values. To quantify this for DNNs, the study focuses on two primary metrics:

  1. Classification Margin: The minimum distance from a training sample to the decision boundary in the input space.
  2. Jacobian Matrix ($J(x)$): The matrix of first-order partial derivatives of the network's output with respect to its input. Its spectral norm measures how much the network amplifies small changes in the input.

The core mathematical contribution is the proof that the classification margin is lower-bounded by the ratio of the network's output score to the spectral norm of its Jacobian. Crucially, the resulting generalization bound depends on the intrinsic complexity of the data (its covering number) and the Jacobian's norm, but remains independent of the number of layers ($L$) or neurons. This explains why deep networks can generalize effectively if their sensitivity to input perturbations is controlled.

The Jacobian Regularizer

While the spectral norm provides the tightest theoretical bound, its direct optimization is computationally prohibitive due to the need for Singular Value Decomposition (SVD). To bridge theory and practice, the authors propose a surrogate regularizer based on the Frobenius norm of the Jacobian ($R_F(W)$).

To further enhance efficiency for large-scale architectures, a "per-layer" approximation is introduced. This version penalizes the norm of individual weight matrices, effectively bounding the product of norms that constitutes the Jacobian of the entire network.

Experimental Validation

The proposed regularizer was evaluated against standard $L_2$ regularization (weight decay) across several benchmarks:

  • MNIST & CIFAR-10: Jacobian regularization consistently outperformed weight decay. Notably, on MNIST, a 4-layer network using this regularizer achieved the same performance with 20,000 samples as a standard network did with 50,000 samples, demonstrating significant data efficiency.
  • ImageNet (ILSVRC2012): Using ResNet-18, the authors tested the regularizer on a large-scale task. Without standard data augmentation, the generalization gap was reduced from approximately 61% to 23%.
  • Computational Efficiency: While the full Jacobian calculation is intensive, the per-layer approximation added only about 58% overhead to ImageNet training batches, making it a viable alternative for modern deep learning pipelines.

Key Insights

  • Depth Independence: The analysis confirms that depth is not an inherent barrier to generalization. Provided the Jacobian norm is bounded, a 100-layer network can generalize as effectively as a shallow one.
  • Theoretical Support for Normalization: The paper provides a formal justification for the success of Batch Normalization and Weight Normalization. These techniques implicitly constrain the Jacobian by normalizing weight rows, thereby preserving the classification margin.
  • ResNets as Ensembles: The study of Residual Networks (ResNets) reveals that their Jacobian is essentially a sum of Jacobians of all possible sub-paths, supporting the hypothesis that ResNets function as ensembles of shallower networks.

Conclusion

By focusing on the Jacobian matrix, this work provides a robust theoretical foundation for the generalization capabilities of deep neural networks. The proposed Jacobian regularizer offers a principled way to improve model performance and stability, moving beyond simple parameter-counting toward a more nuanced understanding of network geometry.

Summary updated

Abstract

The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization re-parametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED and ImageNet datasets.
BibTeX
@article{sokolic2017margin,
  author = {Jure Sokolic and Raja Giryes and Guillermo Sapiro and Miguel R. D. Rodrigues},
  title = {{Margin Preservation of Deep Neural Networks}},
  journal = {IEEE Trans. Signal Process.},
  year = {2017},
  doi = {10.1109/tsp.2017.2708039},
}
Figure