Introduction
Most common regularization techniques in deep learning, such as L1 or L2 weight decay, primarily focus on the network's parameters. While some methods target internal representations (activations), they typically treat all data samples as a single group. This "class-blind" approach can be a missed opportunity for classification tasks, as internal representations of different classes often overlap significantly even after training, making it harder for final layers to distinguish between them. This paper introduces two novel "class-wise" representation regularizers, cw-CR and cw-VR, designed to explicitly organize internal activations based on class labels.
Proposed Class-wise Regularizers
The authors propose adding penalty terms to the standard loss function to shape the geometry of hidden representations:
- cw-CR (Class-wise Covariance Regularizer): This targets the off-diagonal elements of the covariance matrix for activations within each class. By minimizing these elements, the model learns features that are statistically independent for samples of the same class, reducing redundant co-adaptation between neurons.
- cw-VR (Class-wise Variance Regularizer): This minimizes the variance of each unit's activations for samples within the same class. The goal is "feature compactness," ensuring that all samples of a specific class produce highly similar activation patterns.
These regularizers are computationally efficient as they utilize the statistics of the current mini-batch rather than requiring complex pair-wise distance calculations.
Visualizing Representation Shaping
Experimental results on MNIST and CIFAR datasets demonstrate that these regularizers significantly alter the internal structure of the network. As shown in the visualizations below, while baseline models exhibit messy, overlapping activation clusters, class-wise regularizers—particularly cw-VR—produce distinct, tightly packed, and elongated shapes for each class.
Key Insights and Results
Layer-wise Sensitivity
A critical finding is that regularizing the upper layers (closer to the output) is far more effective than regularizing lower layers. Lower layers must represent general concepts like edges or textures shared across classes; forcing them into class-specific shapes can be counterproductive. As seen in the test error plots, performance gains are most pronounced when applying regularization to the final fully connected or convolutional layers.
Performance Gains
The proposed regularizers improved performance over the baseline and standard weight decay in 21 out of 22 tested scenarios. Notably, on CIFAR-10 using ResNet-110, cw-VR reduced the error rate from 6.43% to 6.00%. The variance regularizer (cw-VR) often emerged as the top performer, creating "needle-like" distributions in high-dimensional space that are easily separable by the final classification layer.
Conclusion
By explicitly using label information to shape internal representations, cw-CR and cw-VR provide a powerful and efficient alternative to parameter-based regularization. While the methods show sensitivity to the penalty weight hyperparameter and can lead to "activation squashing" (where the network makes activations very small to minimize the penalty), the resulting statistical benefits of independence and compactness consistently drive better generalization across various architectures.
Summary updated