In standard neural network classifiers, the final layer computes a score for each class using a dot product between a feature vector $x$ and a weight vector $w$. Mathematically, this is expressed as $w^T x = \|w\| \|x\| \cos(\theta)$. While effective for general tasks, this formulation introduces a hidden vulnerability: the classification score is sensitive to the magnitudes of the weights and features, not just their semantic direction.
Research across incremental learning and long-tailed recognition has identified that these magnitudes often become proxies for class frequency or temporal recency, leading to severe model bias.
The Magnitude Bias Problem
In scenarios where data is imbalanced or arrives sequentially, the norms of the weight vectors ($\|w\|$) tend to diverge. In long-tailed datasets, classes with more samples (head classes) receive more frequent gradient updates, causing their weight norms to grow significantly larger than those of rare (tail) classes. Similarly, in Class-Incremental Learning (CIL), the weights for newly introduced classes often exhibit much larger magnitudes than those of older classes.
As Hou et al. and Zhao et al. observe, this creates a "rich-get-richer" effect. Because the dot product is proportional to $\|w\|$, the model produces higher confidence scores for classes with larger weight norms, regardless of the actual feature similarity. This bias forces the decision boundary to shrink the space allocated to minority or older classes, leading to "catastrophic forgetting" or the suppression of rare categories.
Projecting to the Hypersphere
To solve this, researchers have proposed replacing the standard linear layer with a cosine classifier (or cosine normalization). By L2-normalizing both the feature vector $x$ and the weight vector $w$, the classification score is reduced to the cosine of the angle between them:
$$Score = \frac{w^T x}{\|w\| \|x\|} = \cos(\theta)$$
This architectural change forces all class embeddings onto a shared high-dimensional hypersphere. As Hou et al. argue, this neutralizes the bias caused by varying weight magnitudes, ensuring that classification decisions are based purely on the orientation (semantic content) of the features rather than the scale of the weights. Lei et al. further note that this "rectified" normalization prevents the model from incorrectly categorizing old samples into newer, more dominant categories simply because the new weights are larger.
Balancing Incremental Learning
In incremental learning, the bias toward new classes is so predictable that it can be corrected even after training. Zhao et al. introduced weight norm alignment, a zero-parameter method that rescales the weights of the final layer so that the average norm of new class weights matches that of old classes.
Similarly, He and Zhu and Huang et al. demonstrate that normalizing output logits by the L2 norm of the weight vectors serves as a computationally inexpensive post-hoc adjustment. This ensures that classification confidence is based on feature relevance rather than the temporal order in which classes were learned. He et al. highlight that this connection is fundamental: class-incremental learning is essentially a severe class-imbalance problem where the "new" classes act as the majority.
Taming the Long Tail
In long-tailed object detection and segmentation, the goal is to prevent "head" classes from overwhelming "tail" classes. Wang et al. and Hu et al. employ scaled cosine similarity to decouple scores from weight magnitudes. This prevents high-frequency classes from dominating the optimization process through high-magnitude gradients.
To further refine this, Wang and Yu proposed the C2AM Loss, which combines a cosine classifier with an adaptive angular margin. By adjusting the margin based on weight norm ratios, the authors provide additional protection for rare classes, ensuring they maintain a distinct region in the feature space.
Generalization Beyond Classification
The benefits of magnitude invariance extend to other domains where data distributions are non-stationary or sparse:
- Zero-Shot Learning: Liu et al. show that cosine metric learning improves generalization to unseen categories by bounding the variance of neural activations.
- Few-Shot Learning: Huang et al. found that normalizing class prototypes to unit vectors prevents word-frequency bias in Named Entity Recognition.
- Adversarial Robustness: Liu et al. argue that cosine-based classifiers reduce vulnerability to adversarial attacks that exploit feature scaling to cross decision boundaries.
By shifting the focus from "how much" a weight was updated to "in what direction" it points, cosine normalization provides a robust foundation for models that must learn continuously and equitably.
Key Foundational Work
Learning a Unified Classifier Incrementally via Rebalancing. (2019) — contributed 2 concept elements to this direction.
Dynamic Few-Shot Visual Learning Without Forgetting (2018) — contributed 2 concept elements to this direction.
Continuous Learning for Large-scale Personalized Domain Classification (2019) — contributed 2 concept elements to this direction.