Introduction
In deep learning, managing "internal covariate shift" is a fundamental challenge. Standard techniques like Batch Normalization (BN) and Layer Normalization (LN) have become ubiquitous because they control the first two moments—mean and variance—of activations. However, these methods do not explicitly constrain the shape of the activation distribution.
This paper argues that the specific distribution shape is critical. From an information-theoretic perspective, the Gaussian distribution is unique: it maximizes representational capacity for a fixed variance and offers superior robustness to random noise. Furthermore, in a Gaussian regime, lack of correlation implies independence, which helps prevent redundant feature co-adaptation. The authors investigate whether explicitly forcing internal activations to follow a Gaussian distribution improves model performance.
Normality Normalization (NN)
The authors propose Normality Normalization (NN) as a modular upgrade to existing normalization layers. NN introduces two primary steps before the final affine transformation (the $\gamma$ and $\beta$ parameters):
- The Power Transform: The Yeo-Johnson transform is employed to "Gaussianize" the data. To overcome the computational overhead of finding the optimal transform parameter ($\lambda$), the authors developed a fast, one-step approximation using the Newton-Raphson method based on a quadratic expansion of the likelihood function.
- Scaled Additive Gaussian Noise: During training, Gaussian noise is injected, scaled by the $\ell_1$-norm of the current minibatch. This ensures that regularization remains proportional to the signal strength.
NN is designed as a "drop-in" replacement; for instance, "BatchNormalNorm" (BNN) replaces standard BatchNorm, and "LayerNormalNorm" (LNN) replaces LayerNorm.
Experimental Setup
The researchers evaluated NN across a diverse set of benchmarks and architectures:
- Datasets: SVHN, CIFAR10, CIFAR100, Food101, ImageNet, STL10, and Caltech101.
- Architectures: Vision Transformers (ViT), ResNet variants (RN18, RN34, RN50), and WideResNets.
- Conditions: Performance was tested both with and without standard data augmentations to isolate the impact of the normalization layer.
Analysis of Gaussianity
A key insight of the paper is that while standard networks may start with Gaussian-like activations at initialization, they quickly lose this property during training. Normality Normalization is specifically designed to counteract this drift and maintain Gaussianity throughout the entire training process.
As shown in the figure above, standard BatchNorm (orange) exhibits significant fluctuations and a general decline in normality ($R^2$) across layers. In contrast, BatchNormalNorm (blue) maintains a near-perfect Gaussian distribution throughout the network. This enforcement of normality increases joint independence between channel features, leading to more diverse and less redundant representations. This effect is particularly pronounced in narrower networks, where NN acts as a corrective force for finite-width models that naturally drift away from Gaussian behavior.
Performance Comparison
The empirical results demonstrate consistent and often substantial gains across various tasks and architectures.
Key quantitative findings include:
- Vision Transformers: On the Food101 dataset, LNN achieved 79.11% accuracy, a significant jump from the 73.25% achieved by standard LayerNorm.
- ResNets: On STL10, a ResNet34 using BNN reached 63.86% accuracy, outperforming the standard BatchNorm baseline of 58.82%.
- Robustness: BNN models demonstrated lower relative error in noise-injection tests at inference time, suggesting they are fundamentally more stable than their BN counterparts.
Discussion and Future Work
While Normality Normalization provides clear performance benefits, it introduces some computational overhead due to the logarithmic and power operations in the Yeo-Johnson transform. Although the Newton-Raphson approximation mitigates this, training runtimes are measurably higher.
Future research directions include developing even more efficient mathematical approximations for the transforms and exploring the impact of "Gaussianizing" model weights in addition to activations. Furthermore, while the method improves robustness to random noise, its effectiveness against targeted adversarial attacks remains an open question.
Summary updated