Improve the Robustness and Accuracy of Deep Neural Network with L2, ∞ Normalization

Lijia Yu, Xiao-Shan GaoarXiv 2020 · OpenReview

Deep Neural Networks (DNNs) have revolutionized machine learning, yet they remain notoriously "brittle." Despite state-of-the-art performance, these models often suffer from overfitting—learning noise rather than patterns—and a lack of robustness, where imperceptible input perturbations can trigger incorrect predictions. This paper introduces a geometric and theoretical framework to address these vulnerabilities using $L_{2,\infty}$ normalization.

The $L_{2,\infty}$ Normalization Framework

Standard regularization techniques like $L_1$ or $L_2$ (Frobenius) norms constrain the total weight magnitude but do not explicitly account for the geometric structure of the functions learned by ReLU-based networks. The authors propose $L_{2,\infty}$ normalization, defined as the maximum $L_2$ norm among all rows of a weight matrix $M$.

Implementation is straightforward: the authors modify standard gradient descent with a projection step. After each weight update, if the $L_2$ norm of any row in a weight matrix exceeds a pre-defined constant $c$, that row is scaled back down to $c$. This ensures that no single neuron (represented by a row) exerts an outsized influence, leading to a more balanced and stable feature representation.

Geometric Smoothing and Dihedral Angles

A key contribution of this work is the geometric analysis of ReLU-based DNNs. These networks represent piecewise linear functions whose graphs are complex polyhedra in high-dimensional space. The authors prove that $L_{2,\infty}$ normalization directly influences the "smoothness" of these functions by forcing the dihedral angles between adjacent faces of the polyhedron to be larger (flatter).

Geometric representation of dihedral angles in relation to weight norms.

As illustrated in the mathematical models above, constraining the row-wise norms prevents the network from making sharp, erratic fluctuations in its decision boundaries. This geometric smoothing is essential for both generalization and resistance to adversarial noise.

Theoretical Robustness and Complexity

Beyond geometric intuition, the paper provides rigorous mathematical guarantees:

  • Robustness Metric: The authors propose the average radius of "maximal robust spheres" centered at training data points. Within these spheres, the network's classification is guaranteed to remain constant. They derive a mathematical lower bound for this "robust radius" based on weight norms.
  • Rademacher Complexity: The paper establishes a new upper bound for Rademacher complexity specifically for $L_{2,\infty}$ normalized networks. This complexity measure quantifies the model's ability to fit random noise, providing a theoretical basis for the observed improvements in generalization.

Empirical Results

The effectiveness of $L_{2,\infty}$ normalization was validated on two benchmark datasets:

  1. MNIST (Robustness): When subjected to increasing levels of Gaussian noise, $L_{2,\infty}$ normalized models consistently maintained higher accuracy than baseline models or those using $L_1$ regularization.
  2. CIFAR-10 (Generalization): The method significantly reduced overfitting. A model with $L_{2,\infty}$ normalization (threshold $c=0.5$) achieved 74.52% accuracy, outperforming the standard baseline (72.21%) and various $L_1/L_2$ configurations.

Experiments also revealed an optimal "sweet spot" for the constraint threshold $c$. If $c$ is too restrictive, the network lacks the capacity to learn complex patterns; if it is too loose, the beneficial smoothing effect is lost.

Conclusion and Future Directions

$L_{2,\infty}$ normalization offers a principled way to improve DNN reliability by bridging the gap between row-wise weight constraints and the geometric smoothness of the learned function. While currently focused on fully connected layers, future work aims to extend this analysis to Convolutional Neural Networks (CNNs) and explore higher-order smoothness constraints to further refine network behavior.

Summary updated

Abstract

In this paper, the robustness and accuracy of the deep neural network (DNN) was enhanced by introducing the $L_{2,\infty}$ normalization of the weight matrices of the DNN with Relu as the activation function. It is proved that the $L_{2,\infty}$ normalization leads to large dihedral angles between two adjacent faces of the polyhedron graph of the DNN function and hence smoother DNN functions, which reduces over-fitting. A measure is proposed for the robustness of a classification DNN, which is the average radius of the maximal robust spheres with the sample data as centers. A lower bound for the robustness measure is given in terms of the $L_{2,\infty}$ norm. Finally, an upper bound for the Rademacher complexity of DNN with $L_{2,\infty}$ normalization is given. An algorithm is given to train a DNN with the $L_{2,\infty}$ normalization and experimental results are used to show that the $L_{2,\infty}$ normalization is effective to improve the robustness and accuracy.
BibTeX
@misc{yu2020improve,
  author = {Lijia Yu and Xiao-Shan Gao},
  title = {{Improve the Robustness and Accuracy of Deep Neural Network with L2, ∞ Normalization}},
  year = {2020},
  eprint = {2010.04912},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.04912},
}
Figure