TL;DR
While traditional regularization techniques like weight decay focus on shrinking model weights to prevent overfitting, Feature Incay does the exact opposite for representations: it encourages larger feature norms. By "stretching" feature vectors, this method improves class separability and naturally upweights hard-to-classify examples, leading to consistent performance gains across standard computer vision benchmarks.
The Problem: The Overlooked Feature Norm
In deep multi-class classification, models typically rely on the Softmax loss. Mathematically, the score for a given class is the dot product of a weight vector ($w$) and a feature vector ($f$), which can be expressed as:
$$z = \|w\| \|f\| \cos(\theta)$$
Historically, most research has focused on the angle ($\theta$)—attempting to make classes more angularly separable—or on shrinking the weights ($w$) via weight decay to prevent overfitting. The magnitude of the feature vector ($\|f\|$), however, is often treated as a mere byproduct or intentionally ignored by normalizing it to 1.
The authors of this paper observed a critical, overlooked trend: samples with larger feature norms are significantly more likely to be classified correctly. Conversely, misclassified "hard" samples often have tiny norms. Because their norms are small, they contribute very little to the gradient during training. This creates a "suppression" effect where the model effectively stops learning from the very samples it finds most difficult.
The Solution: Feature Incay
To combat this suppression effect, the authors introduce Feature Incay, a regularization term that explicitly penalizes small feature norms. While weight decay "decays" weights toward zero, feature incay "stretches" features away from zero.
The specific implementation is called the Reciprocal Norm Loss:
$$\mathcal{L}_{incay} = \lambda \frac{1}{N} \sum_{i=1}^{N} \frac{1}{\|f_i\|^2 + \epsilon}$$
- Dynamic Growth: Unlike previous methods that fix the feature norm to a constant (such as $L_2$-softmax), Feature Incay allows the norm to grow dynamically in the original feature space.
- The Reciprocal Advantage: The gradient of this reciprocal loss is much stronger for small norms than for large ones. This creates an adaptive effect: it aggressively pushes "laggard" features (those with small norms) to grow, while leaving already well-defined, large-norm features relatively alone. This naturally reduces intra-class variance, making clusters more compact.
Geometric and Training Implications
Feature Incay introduces several powerful dynamics to the learning process:
- The "Stretch" vs. "Shrink" Duality: Weight decay prevents the model from becoming too complex by shrinking weights. Feature Incay prevents the model from becoming "lazy" by ensuring representations are prominent and distinct.
- Implicit Hard Example Mining: Because misclassified samples typically have small norms, the reciprocal loss function naturally assigns them higher gradients. This forces the model to focus on these "hard" samples without needing complex, computationally expensive data sampling heuristics.
- Geometric Separability: From a computational standpoint, larger feature norms lead to larger score differences between the correct and incorrect categories. Geometrically, this pushes the decision boundaries further apart, creating a more robust margin.
- Orthogonality to Angular Margins: Feature Incay can be seamlessly combined with angular margin losses (like L-Softmax or A-Softmax). While those methods optimize the direction of features, Feature Incay optimizes the magnitude, providing a complementary boost to overall performance.
Key Results
The effectiveness of Feature Incay was validated on four widely-used computer vision benchmarks: MNIST, CIFAR-10, CIFAR-100, and Labeled Faces in the Wild (LFW).
The authors demonstrated that adding Feature Incay (via Reciprocal Norm Loss) to standard baselines consistently lowered error rates:
- CIFAR-10: Standard Softmax error dropped from 8.59% to 7.84%.
- CIFAR-100: Combining RN with Large-Margin Softmax (L-Softmax) reduced the error from 29.95% to 29.18%.
- LFW Face Verification: The combination of RN and A-Softmax achieved a state-of-the-art accuracy of 99.47%.
Visualizing the training dynamics confirms the theory: applying Feature Incay steadily increases the average feature norm across classes while simultaneously driving down loss and boosting test accuracy.
Limitations and Future Work
While promising, Feature Incay comes with a few considerations:
- Hyperparameter Tuning: The method introduces a new hyperparameter, $\lambda$. If $\lambda$ is too large, it can dominate the Softmax loss and hinder convergence. A relatively small weight (e.g., 0.1 or 0.01) is usually optimal.
- Upper Bounds: The paper proves there is a theoretical upper bound to how much the norm should grow to maintain compactness, but the current loss doesn't explicitly enforce a ceiling. It relies on the balance between the Softmax loss and weight decay to keep growth in check.
- Feature Scaling vs. Learning: Simply multiplying features by a large constant (scaling) doesn't work and can cause gradients to explode. The learning must happen organically through the backpropagation of the reciprocal loss to remain stable. Future work could explore more complex functional forms for this "stretching" effect.
Summary updated