Feature Incay for Representation Regularization

Yuhui Yuan, Kuiyuan Yang, Jianyuan Guo, Chao Zhang, Jingdong Wang

TL;DR

While traditional regularization techniques like weight decay focus on shrinking model weights to prevent overfitting, Feature Incay does the exact opposite for representations: it encourages larger feature norms. By "stretching" feature vectors, this method improves class separability and naturally upweights hard-to-classify examples, leading to consistent performance gains across standard computer vision benchmarks.

The Problem: The Overlooked Feature Norm

In deep multi-class classification, models typically rely on the Softmax loss. Mathematically, the score for a given class is the dot product of a weight vector ($w$) and a feature vector ($f$), which can be expressed as:

$$z = \|w\| \|f\| \cos(\theta)$$

Historically, most research has focused on the angle ($\theta$)—attempting to make classes more angularly separable—or on shrinking the weights ($w$) via weight decay to prevent overfitting. The magnitude of the feature vector ($\|f\|$), however, is often treated as a mere byproduct or intentionally ignored by normalizing it to 1.

The authors of this paper observed a critical, overlooked trend: samples with larger feature norms are significantly more likely to be classified correctly. Conversely, misclassified "hard" samples often have tiny norms. Because their norms are small, they contribute very little to the gradient during training. This creates a "suppression" effect where the model effectively stops learning from the very samples it finds most difficult.

Accuracy and sample count binned by feature L2-norm.

The Solution: Feature Incay

To combat this suppression effect, the authors introduce Feature Incay, a regularization term that explicitly penalizes small feature norms. While weight decay "decays" weights toward zero, feature incay "stretches" features away from zero.

The specific implementation is called the Reciprocal Norm Loss:

$$\mathcal{L}_{incay} = \lambda \frac{1}{N} \sum_{i=1}^{N} \frac{1}{\|f_i\|^2 + \epsilon}$$

  • Dynamic Growth: Unlike previous methods that fix the feature norm to a constant (such as $L_2$-softmax), Feature Incay allows the norm to grow dynamically in the original feature space.
  • The Reciprocal Advantage: The gradient of this reciprocal loss is much stronger for small norms than for large ones. This creates an adaptive effect: it aggressively pushes "laggard" features (those with small norms) to grow, while leaving already well-defined, large-norm features relatively alone. This naturally reduces intra-class variance, making clusters more compact.

Geometric and Training Implications

Feature Incay introduces several powerful dynamics to the learning process:

  • The "Stretch" vs. "Shrink" Duality: Weight decay prevents the model from becoming too complex by shrinking weights. Feature Incay prevents the model from becoming "lazy" by ensuring representations are prominent and distinct.
  • Implicit Hard Example Mining: Because misclassified samples typically have small norms, the reciprocal loss function naturally assigns them higher gradients. This forces the model to focus on these "hard" samples without needing complex, computationally expensive data sampling heuristics.
  • Geometric Separability: From a computational standpoint, larger feature norms lead to larger score differences between the correct and incorrect categories. Geometrically, this pushes the decision boundaries further apart, creating a more robust margin.
  • Orthogonality to Angular Margins: Feature Incay can be seamlessly combined with angular margin losses (like L-Softmax or A-Softmax). While those methods optimize the direction of features, Feature Incay optimizes the magnitude, providing a complementary boost to overall performance.

Key Results

The effectiveness of Feature Incay was validated on four widely-used computer vision benchmarks: MNIST, CIFAR-10, CIFAR-100, and Labeled Faces in the Wild (LFW).

The authors demonstrated that adding Feature Incay (via Reciprocal Norm Loss) to standard baselines consistently lowered error rates:

  • CIFAR-10: Standard Softmax error dropped from 8.59% to 7.84%.
  • CIFAR-100: Combining RN with Large-Margin Softmax (L-Softmax) reduced the error from 29.95% to 29.18%.
  • LFW Face Verification: The combination of RN and A-Softmax achieved a state-of-the-art accuracy of 99.47%.
Training dynamics on CIFAR-10 showing increased average feature norm and improved accuracy with Reciprocal Norm Loss.

Visualizing the training dynamics confirms the theory: applying Feature Incay steadily increases the average feature norm across classes while simultaneously driving down loss and boosting test accuracy.

Limitations and Future Work

While promising, Feature Incay comes with a few considerations:

  • Hyperparameter Tuning: The method introduces a new hyperparameter, $\lambda$. If $\lambda$ is too large, it can dominate the Softmax loss and hinder convergence. A relatively small weight (e.g., 0.1 or 0.01) is usually optimal.
  • Upper Bounds: The paper proves there is a theoretical upper bound to how much the norm should grow to maintain compactness, but the current loss doesn't explicitly enforce a ceiling. It relies on the balance between the Softmax loss and weight decay to keep growth in check.
  • Feature Scaling vs. Learning: Simply multiplying features by a large constant (scaling) doesn't work and can cause gradients to explode. The learning must happen organically through the backpropagation of the reciprocal loss to remain stable. Future work could explore more complex functional forms for this "stretching" effect.

Summary updated

Abstract

Softmax-based loss is widely used in deep learning for multi-class classification, where each class is represented by a weight vector and each sample is represented as a feature vector. Different from traditional learning algorithms where features are pre-defined and only weight vectors are tunable through training, feature vectors are also tunable as representation learning in deep learning. Thus we investigate how to improve the classification performance by better adjusting the features. One main observation is that elongating the feature norm of both correctly-classified and mis-classified feature vectors improves learning: (1) increasing the feature norm of correctly-classified examples induce smaller training loss; (2) increasing the feature norm of mis-classified examples can upweight the contribution from hard examples. Accordingly, we propose feature incay to regularize representation learning by encouraging larger feature norm. In contrast to weight decay which shrinks the weight norm, feature incay is proposed to stretch the feature norm. Extensive empirical results on MNIST, CIFAR10, CIFAR100 and LFW demonstrate the effectiveness of feature incay.
BibTeX
@inproceedings{yuan2018feature,
  author = {Yuhui Yuan and Kuiyuan Yang and Jianyuan Guo and Chao Zhang and Jingdong Wang},
  title = {{Feature Incay for Representation Regularization}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
}
Figure