Utilizing Class Information for Deep Network Representation Shaping

Daeyoung Choi, Wonjong Rhee

Introduction

Most common regularization techniques in deep learning, such as L1 or L2 weight decay, primarily focus on the network's parameters. While some methods target internal representations (activations), they typically treat all data samples as a single group. This "class-blind" approach can be a missed opportunity for classification tasks, as internal representations of different classes often overlap significantly even after training, making it harder for final layers to distinguish between them. This paper introduces two novel "class-wise" representation regularizers, cw-CR and cw-VR, designed to explicitly organize internal activations based on class labels.

Proposed Class-wise Regularizers

The authors propose adding penalty terms to the standard loss function to shape the geometry of hidden representations:

  1. cw-CR (Class-wise Covariance Regularizer): This targets the off-diagonal elements of the covariance matrix for activations within each class. By minimizing these elements, the model learns features that are statistically independent for samples of the same class, reducing redundant co-adaptation between neurons.
  2. cw-VR (Class-wise Variance Regularizer): This minimizes the variance of each unit's activations for samples within the same class. The goal is "feature compactness," ensuring that all samples of a specific class produce highly similar activation patterns.

These regularizers are computationally efficient as they utilize the statistics of the current mini-batch rather than requiring complex pair-wise distance calculations.

Visualizing Representation Shaping

Experimental results on MNIST and CIFAR datasets demonstrate that these regularizers significantly alter the internal structure of the network. As shown in the visualizations below, while baseline models exhibit messy, overlapping activation clusters, class-wise regularizers—particularly cw-VR—produce distinct, tightly packed, and elongated shapes for each class.

Visual comparison of internal activations (histograms, scatter plots, and 3D PCA) across different regularization methods.

Key Insights and Results

Layer-wise Sensitivity

A critical finding is that regularizing the upper layers (closer to the output) is far more effective than regularizing lower layers. Lower layers must represent general concepts like edges or textures shared across classes; forcing them into class-specific shapes can be counterproductive. As seen in the test error plots, performance gains are most pronounced when applying regularization to the final fully connected or convolutional layers.

Test error rates across different network layers for MNIST and CIFAR-100, showing the effectiveness of regularizing deeper layers.

Performance Gains

The proposed regularizers improved performance over the baseline and standard weight decay in 21 out of 22 tested scenarios. Notably, on CIFAR-10 using ResNet-110, cw-VR reduced the error rate from 6.43% to 6.00%. The variance regularizer (cw-VR) often emerged as the top performer, creating "needle-like" distributions in high-dimensional space that are easily separable by the final classification layer.

Conclusion

By explicitly using label information to shape internal representations, cw-CR and cw-VR provide a powerful and efficient alternative to parameter-based regularization. While the methods show sensitivity to the penalty weight hyperparameter and can lead to "activation squashing" (where the network makes activations very small to minimize the penalty), the resulting statistical benefits of independence and compactness consistently drive better generalization across various architectures.

Summary updated

Abstract

Statistical characteristics of deep network representations, such as sparsity and correlation, are known to be relevant to the performance and interpretability of deep learning. When a statistical characteristic is desired, often an adequate regularizer can be designed and applied during the training phase. Typically, such a regularizer aims to manipulate a statistical characteristic over all classes together. For classification tasks, however, it might be advantageous to enforce the desired characteristic per class such that different classes can be better distinguished. Motivated by the idea, we design two class-wise regularizers that explicitly utilize class information: class-wise Covariance Regularizer (cw-CR) and classwise Variance Regularizer (cw-VR). cw-CR targets to reduce the covariance of representations calculated from the same class samples for encouraging feature independence. cw-VR is similar, but variance instead of covariance is targeted to improve feature compactness. For the sake of completeness, their counterparts without using class information, Covariance Regularizer (CR) and Variance Regularizer (VR), are considered together. The four regularizers are conceptually simple and computationally very efficient, and the visualization shows that the regularizers indeed perform distinct representation shaping. In terms of classification performance, significant improvements over the baseline and L1/L2 weight regularization methods were found for 21 out of 22 tasks over popular benchmark datasets. In particular, cw-VR achieved the best performance for 13 tasks including ResNet-32/110.
BibTeX
@inproceedings{choi2019utilizing,
  author = {Daeyoung Choi and Wonjong Rhee},
  title = {{Utilizing Class Information for Deep Network Representation Shaping}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33013396},
}

Loading related content...

Figure