Abstract
Deep neural networks (DNNs) often exhibit remarkable generalization capabilities despite being significantly over-parameterized. While theoretical frameworks suggest that model capacity is governed by weight norms rather than parameter counts, these theories typically assume clean training data. This paper investigates the behavior of various regularization techniques—including $L_2$ weight decay, Dropout, Jacobian regularization, and Spectral Normalization—when training data is corrupted by input noise. Our findings indicate that deep architectures possess an inherent stability to noise, and that Spectral Normalization and $L_2$ regularization are particularly effective at maintaining high accuracy while controlling model complexity in noisy regimes.
Introduction
The success of deep learning is often attributed to the ability of over-parameterized models to generalize without overfitting. Traditional learning theory, such as Vapnik-Chervonenkis (VC) dimension and Rademacher complexity, provides bounds on generalization error based on model capacity. In the context of neural networks, this capacity is frequently linked to the norm of the weights. However, real-world data is rarely pristine. A critical question remains: how do standard regularization methods perform when the input signal is noisy? This study aims to bridge this gap by evaluating how different regularizers influence both the performance and the internal complexity measures of networks under varying levels of Gaussian input noise.
Methodology
The researchers evaluated several regularization schemes across a variety of datasets and architectures. The regularizers included $L_2$ weight decay, Dropout, Jacobian regularization, Spectral Normalization, and newer distribution-dependent regularizers like DARC1 and LCNN. The complexity of the resulting models was quantified using several theoretical measures and bounds.
Experiments were conducted on non-image datasets (Adult, Codrna, Covertype) and image datasets (MNIST, CIFAR10). Architectures ranged from shallow convolutional networks to deep Wide ResNet 28-10 models. Gaussian noise was introduced to the training sets, while test sets remained clean to measure generalization performance.
Results and Discussion
The experimental results reveal several key insights into the interaction between noise and regularization.
Noise Stability in Deep Networks
A primary observation is that deeper architectures are naturally more robust to input noise than shallow ones. This suggests that deep networks may perform an implicit form of noise reduction as signals propagate through successive layers.
Performance of Regularizers
Spectral Normalization emerged as the most robust technique, consistently achieving the highest test accuracies. By bounding the Lipschitz constant, it prevents the network from becoming overly sensitive to input fluctuations. In contrast, Dropout performed poorly in noisy environments, failing to effectively control model complexity and leading to unstable performance as noise levels increased.
As illustrated in Figure 1, while accuracies for shallow networks generally decline with increased noise, Jacobian regularization and Spectral Normalization mitigate this drop more effectively than other methods. Furthermore, $L_2$ regularization proved to be a strong proxy for complexity; networks regularized with $L_2$ learned "simpler" hypotheses that generalized better, even when trained on noisy data.
Conclusion
This study highlights the critical role of regularization in maintaining model robustness against input noise. Spectral Normalization and $L_2$ regularization are identified as superior tools for balancing accuracy and complexity. Future work will extend this analysis to other noise types, such as adversarial or label noise, and explore recurrent architectures like LSTMs.
Summary updated