On the Stability of Neural Networks in Deep Learning

Blaise DelattrearXiv 2025 · DOI · arXiv

Abstract

Deep learning models are often characterized by a lack of stability, manifesting as vulnerability to adversarial perturbations, optimization difficulties, and poor generalization to unseen data. This thesis by Blaise Delattre proposes a unified framework to address these issues by controlling the sensitivity of neural networks to both inputs and parameters. Key contributions include the Gram Iteration for efficient spectral norm estimation, Spectral Rescaling for Lipschitz-constrained architectures, and Activation Decay for promoting flat minima. Experimental results across computer vision and natural language processing benchmarks demonstrate significant improvements in robustness, training efficiency, and generalization.

Introduction

The "brittleness" of deep neural networks remains a significant barrier to their deployment in safety-critical applications. Adversarial examples—small, often imperceptible changes to inputs that lead to incorrect outputs—highlight a fundamental lack of input stability. Simultaneously, the complexity of the loss landscape can lead to training instability and poor generalization. This research argues that these disparate problems can be addressed through a common lens of stability analysis. By mathematically grounding architectural choices and regularization techniques, we can develop models that are inherently more reliable.

Methodology

The thesis introduces several novel algorithms designed to bake stability into the deep learning pipeline:

Gram Iteration (GI) for Spectral Norm Estimation

Spectral normalization is a popular technique for controlling the Lipschitz constant of a network, but it typically relies on the stochastic Power Iteration method. The author proposes Gram Iteration, a deterministic alternative for computing the spectral norm (the largest singular value) of a matrix. GI provides a strict upper bound at every step and exhibits supergeometric convergence, making it significantly faster and more reliable than traditional methods.

Spectral Rescaling (SR)

To ensure a network is 1-Lipschitz (meaning it does not amplify input noise), the author introduces Spectral Rescaling. This novel layer design acts as a middle ground between strict Spectral Normalization and Almost Orthogonal Layers. It allows for better information preservation while maintaining the safety guarantees required for adversarial robustness.

Class Partitioning Method (CPM)

Randomized smoothing is a powerful technique for certifying robustness, but it suffers from high statistical overhead in multi-class settings. The Class Partitioning Method optimizes this process by grouping less likely classes into "meta-classes." This simple yet effective approach reduces the number of hypotheses to test, making certified robustness scalable to large-scale datasets like ImageNet.

Activation Decay (AD)

To improve generalization, the author introduces Activation Decay, a deterministic regularization term applied to the penultimate layer's activations. AD serves as a surrogate for noise injection, forcing the optimization process toward "flat minima"—regions of the parameter space where small perturbations to weights do not significantly impact performance.

Results

The proposed methods were rigorously tested on benchmarks including CIFAR-10, CIFAR-100, ImageNet-1k, and various NLP tasks using BERT and T5 models.

  • Computational Performance: Gram Iteration achieves numerical convergence in fewer than 12 iterations, whereas Power Iteration often fails to converge even after 2,000 iterations. Additionally, an explicit gradient formulation for spectral norms reduced GPU memory usage by more than 2x on ResNet architectures.
  • Robustness Gains: The synergy between Lipschitz continuity and Randomized Smoothing allowed for certified robustness radii up to 27% larger than baseline techniques. This approach narrows the gap between theoretical "certified" accuracy and empirical robust performance.
  • Generalization in NLP and LLMs: Activation Decay improved BERT's accuracy across seven NLP tasks and boosted the performance of T5-large on the MMMLU benchmark, demonstrating the universal benefit of promoting flatness in the loss landscape.

Conclusion

This thesis provides a mathematically rigorous yet practical toolkit for stabilizing deep learning models. By replacing stochastic components with deterministic surrogates and optimizing architectural constraints, the author demonstrates that stability can be achieved without sacrificing scalability. Future work will focus on extending these guarantees to the non-Lipschitz attention mechanisms found in Transformer architectures and further reducing the "accuracy-robustness trade-off" inherent in constrained models.

Summary updated

Abstract

Deep learning has achieved remarkable success across a wide range of tasks, but its models often suffer from instability and vulnerability: small changes to the input may drastically affect predictions, while optimization can be hindered by sharp loss landscapes. This thesis addresses these issues through the unifying perspective of sensitivity analysis, which examines how neural networks respond to perturbations at both the input and parameter levels. We study Lipschitz networks as a principled way to constrain sensitivity to input perturbations, thereby improving generalization, adversarial robustness, and training stability. To complement this architectural approach, we introduce regularization techniques based on the curvature of the loss function, promoting smoother optimization landscapes and reducing sensitivity to parameter variations. Randomized smoothing is also explored as a probabilistic method for enhancing robustness at decision boundaries. By combining these perspectives, we develop a unified framework where Lipschitz continuity, randomized smoothing, and curvature regularization interact to address fundamental challenges in stability. The thesis contributes both theoretical analysis and practical methodologies, including efficient spectral norm computation, novel Lipschitz-constrained layers, and improved certification procedures.
BibTeX
@misc{delattre2025stability,
  author = {Blaise Delattre},
  title = {{On the Stability of Neural Networks in Deep Learning}},
  year = {2025},
  eprint = {2510.25282},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.25282},
}
Figure