Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

Chunjie Luo, Jianfeng Zhan, Lei Wang, Qiang Yang

Abstract

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks as well as convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.

Introduction

In standard neural network architectures, the "pre-activation" value passed into functions like ReLU is calculated using a dot product between a weight vector ($w$) and an input vector ($x$). Because the dot product is unbounded, the resulting values can exhibit extremely high variance. This high variance makes models hypersensitive to changes in input distribution—a phenomenon known as internal covariate shift.

While techniques like Batch Normalization (BN) are commonly used to mitigate this, they rely on mini-batch statistics which can be unstable in certain contexts, such as small batch sizes or recurrent architectures. This paper proposes Cosine Normalization, a technique that replaces the traditional dot product with cosine similarity, bounding pre-activations between -1 and 1 to reduce variance and improve generalization.

Methodology

The core idea is to fundamentally change the underlying operation of a neural network layer from an unbounded dot product to a bounded similarity metric.

Neural network layer showing weight vector $w$ and input vector $x$.

The authors introduce two main variations:

  1. Cosine Normalization: Instead of $net = w \cdot x$, the operation becomes: $$net_{norm} = \cos \theta = \frac{w \cdot x}{\|w\| \|x\|}$$ This ensures the input to the activation function is always constrained within the $[-1, 1]$ range.

  2. Centered Cosine Normalization (Pearson Correlation Coefficient - PCC): This version centers both the weights and inputs by their respective means ($\mu_w, \mu_x$) before calculating the cosine similarity. This effectively treats the operation as a correlation check between the filter and the input.

Unlike Weight Normalization, which only divides by the norm of the weights, Cosine Normalization divides by the norms of both weights and inputs. Furthermore, unlike Batch or Layer Normalization which normalize the result of the dot product, Cosine Normalization modifies the operation itself. Notably, when using ReLU, the authors found that learnable re-scaling and re-shifting parameters ($\gamma$ and $\beta$) were often unnecessary.

Experimental Evaluation

The method was evaluated across several benchmarks including MNIST, CIFAR-10/100, and SVHN. A particularly notable test case was the 20NEWS GROUP dataset, which involves high-dimensional text classification using TF-IDF features.

Key Insights

  • Magnitude Independence: Cosine normalization makes the network robust to input scaling (e.g., image brightness) because it focuses solely on the angle between weights and inputs.
  • Bounding Variance: Forcing pre-activations into $[-1, 1]$ significantly reduces internal covariate shift, leading to more stable training.
  • High-Dimensional Stability: In high-dimensional spaces, dot product variance is amplified. Cosine similarity provides a much more stable metric for feature matching in these scenarios.

Results

In nearly all experiments, Centered Cosine Normalization (PCC) achieved the lowest test error.

Test error comparison on the 20NEWS dataset, showing PCC outperforming other normalization methods.
  • 20NEWS GROUP: PCC achieved a 29.37% error rate, significantly outperforming Batch Norm's 43.94%. This underscores its efficacy in high-dimensional text data.
  • CIFAR-10: PCC reached 6.39% error, compared to 8.08% for Batch Norm.
  • MNIST: PCC achieved 1.39% error, outperforming Layer Norm (1.43%) and Batch Norm (1.45%).
  • Training Stability: Cosine Normalization exhibited much lower variance in test error across training epochs compared to the fluctuations often seen with Batch Normalization.

Conclusion and Future Work

Cosine Normalization offers a robust alternative to traditional normalization techniques by bounding the fundamental layer operation. While it shows superior performance, particularly in high-dimensional tasks, it does introduce some computational overhead due to Euclidean norm calculations. Future research could investigate its integration with modern architectures like Transformers or deeper Residual Networks, as well as addressing potential issues with zero-vector inputs.

Summary updated

Abstract

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks as well as convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.
BibTeX
@misc{luo2017cosine,
  author = {Chunjie Luo and Jianfeng Zhan and Lei Wang and Qiang Yang},
  title = {{Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks}},
  year = {2017},
  eprint = {1702.05870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.05870},
}
Figure