Abstract
Deep learning models are often characterized by a lack of stability, manifesting as vulnerability to adversarial perturbations, optimization difficulties, and poor generalization to unseen data. This thesis by Blaise Delattre proposes a unified framework to address these issues by controlling the sensitivity of neural networks to both inputs and parameters. Key contributions include the Gram Iteration for efficient spectral norm estimation, Spectral Rescaling for Lipschitz-constrained architectures, and Activation Decay for promoting flat minima. Experimental results across computer vision and natural language processing benchmarks demonstrate significant improvements in robustness, training efficiency, and generalization.
Introduction
The "brittleness" of deep neural networks remains a significant barrier to their deployment in safety-critical applications. Adversarial examples—small, often imperceptible changes to inputs that lead to incorrect outputs—highlight a fundamental lack of input stability. Simultaneously, the complexity of the loss landscape can lead to training instability and poor generalization. This research argues that these disparate problems can be addressed through a common lens of stability analysis. By mathematically grounding architectural choices and regularization techniques, we can develop models that are inherently more reliable.
Methodology
The thesis introduces several novel algorithms designed to bake stability into the deep learning pipeline:
Gram Iteration (GI) for Spectral Norm Estimation
Spectral normalization is a popular technique for controlling the Lipschitz constant of a network, but it typically relies on the stochastic Power Iteration method. The author proposes Gram Iteration, a deterministic alternative for computing the spectral norm (the largest singular value) of a matrix. GI provides a strict upper bound at every step and exhibits supergeometric convergence, making it significantly faster and more reliable than traditional methods.
Spectral Rescaling (SR)
To ensure a network is 1-Lipschitz (meaning it does not amplify input noise), the author introduces Spectral Rescaling. This novel layer design acts as a middle ground between strict Spectral Normalization and Almost Orthogonal Layers. It allows for better information preservation while maintaining the safety guarantees required for adversarial robustness.
Class Partitioning Method (CPM)
Randomized smoothing is a powerful technique for certifying robustness, but it suffers from high statistical overhead in multi-class settings. The Class Partitioning Method optimizes this process by grouping less likely classes into "meta-classes." This simple yet effective approach reduces the number of hypotheses to test, making certified robustness scalable to large-scale datasets like ImageNet.
Activation Decay (AD)
To improve generalization, the author introduces Activation Decay, a deterministic regularization term applied to the penultimate layer's activations. AD serves as a surrogate for noise injection, forcing the optimization process toward "flat minima"—regions of the parameter space where small perturbations to weights do not significantly impact performance.
Results
The proposed methods were rigorously tested on benchmarks including CIFAR-10, CIFAR-100, ImageNet-1k, and various NLP tasks using BERT and T5 models.
- Computational Performance: Gram Iteration achieves numerical convergence in fewer than 12 iterations, whereas Power Iteration often fails to converge even after 2,000 iterations. Additionally, an explicit gradient formulation for spectral norms reduced GPU memory usage by more than 2x on ResNet architectures.
- Robustness Gains: The synergy between Lipschitz continuity and Randomized Smoothing allowed for certified robustness radii up to 27% larger than baseline techniques. This approach narrows the gap between theoretical "certified" accuracy and empirical robust performance.
- Generalization in NLP and LLMs: Activation Decay improved BERT's accuracy across seven NLP tasks and boosted the performance of T5-large on the MMMLU benchmark, demonstrating the universal benefit of promoting flatness in the loss landscape.
Conclusion
This thesis provides a mathematically rigorous yet practical toolkit for stabilizing deep learning models. By replacing stochastic components with deterministic surrogates and optimizing architectural constraints, the author demonstrates that stability can be achieved without sacrificing scalability. Future work will focus on extending these guarantees to the non-Lipschitz attention mechanisms found in Transformer architectures and further reducing the "accuracy-robustness trade-off" inherent in constrained models.
Summary updated