In standard deep learning architectures, the final classification layer is almost always a linear transformation followed by a softmax. Mathematically, the network computes a dot product between a feature vector $x$ and a set of class weights $W$. While effective, this approach harbors a hidden vulnerability: the dot product is sensitive to the magnitude of both the features and the weights.
Research across several domains—from few-shot learning to incremental updates—has converged on a more robust alternative. By replacing the dot product with cosine similarity, researchers have found they can eliminate biases caused by data imbalance, enable instant learning of new categories, and prevent models from "forgetting" old tasks.
The Magnitude Bias
In a standard linear layer, the score for a class is $f_i = W_i^T x$. This score increases if the feature vector $x$ is long or if the weight vector $W_i$ has a large norm. This creates a "rich-get-richer" dynamic.
As Tong Wang and colleagues observe in C2AM Loss: Chasing a Better Decision Boundary for Long-Tail Object Detection, classes with many training examples (head classes) receive more frequent gradient updates, causing their weight norms to grow significantly larger than those of rare (tail) classes. Consequently, the model becomes biased toward head classes simply because their weights are "larger," effectively crushing the signal from rare categories.
Similarly, in incremental learning, Qianli Ma notes in Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models that new tasks often produce features and weights with higher magnitudes, causing the model to overconfidently predict the most recently learned classes.
The Hypersphere Solution
The solution is to decouple the classification decision from the magnitude of the vectors. By applying L2 normalization to both the feature embeddings and the classification weights, the operation transforms into a cosine similarity:
$$ \text{score}_i = \frac{W_i^T x}{\|W_i\| \|x\|} = \cos(\theta_i) $$
This forces all class embeddings and input features onto a shared high-dimensional hypersphere. Because every weight vector now has a unit norm of 1, no single class can dominate the others through sheer volume.
However, cosine values are bounded between -1 and 1, which is too narrow for a standard softmax function to produce sharp probability distributions. To fix this, Hang Qi, Matthew Brown, and David G. Lowe introduced a learnable scaling factor $s$ in Learning with Imprinted Weights. This scale factor stretches the cosine scores back into a range suitable for cross-entropy loss, maintaining the discriminative power of the model while keeping the "fairness" of the angular metric.
Instant Learning through Imprinting
One of the most powerful properties of cosine classifiers is "weight imprinting." In Learning with Imprinted Weights, the authors demonstrate that if you have a single example of a new category, you can simply take its normalized feature embedding and "copy-paste" it directly into the classification weight matrix as a new row.
Because the classifier is based on cosine similarity, this new weight vector acts as a perfect prototype for that class. This allows a network to recognize a new category instantly without any gradient-based fine-tuning, bridging the gap between metric learning (which compares examples) and softmax classification (which uses fixed categories).
Mitigating Catastrophic Forgetting
In incremental learning, models often "forget" old classes because the weights for new classes grow disproportionately large. Saihui Hou and Chen Change Loy show in Learning a Unified Classifier Incrementally via Rebalancing that cosine normalization acts as a natural stabilizer.
By constraining all class embeddings to the hypersphere, the model is forced to treat old and new classes fairly. Timothée Lesort and Irina Rish further emphasize in Continual Learning in Deep Networks: an Analysis of the Last Layer that this simple reparameterization of the output layer can sometimes outperform complex continual learning algorithms. It ensures that the "semantic direction" of a class is preserved, even as the network's internal representations shift to accommodate new data.
Robustness to Imbalance and Outliers
The benefits of this approach extend to medical imaging and security. In RACR-MIL, Anirudh Choudhary uses a cosine-based classifier to grade skin cancer, noting that it mitigates bias in datasets with severe class imbalance by focusing on feature orientation rather than frequency-driven magnitude.
Furthermore, Xiang Xiang and Zihan Zhang demonstrate in Decoupling MaxLogit for Out-of-Distribution Detection that decoupling magnitude from direction improves uncertainty estimation. Standard classifiers often give high-magnitude features high confidence scores, even if they are semantically far from any known class. By focusing on the angular alignment (MaxCosine), the model can more reliably identify out-of-distribution samples that don't fit the directional prototypes of the training data.