PR Product: A Substitute for Inner Product in Neural Networks

Zhennan Wang, Wenbin Zou, Chen XuICCV 2019 · DOI · arXiv

Introduction: The "Lazy Gradient" Problem

In modern deep learning, the inner product between a weight vector ($w$) and a data vector ($x$) is the fundamental building block of fully connected, convolutional, and recurrent layers. However, this operation contains an inherent geometric flaw that can hinder optimization. The gradient of an inner product can be decomposed into two components: a length gradient, which modifies the magnitude of $w$, and a direction gradient, which rotates $w$ to align with $x$.

Mathematically, the direction gradient is proportional to $\sin(\theta)$, where $\theta$ is the angle between $w$ and $x$. As the weight vector becomes more aligned with the data (angle approaches 0 or $\pi$), $\sin(\theta)$ vanishes. This creates a "lazy gradient" effect: the optimizer loses the signal to refine the orientation of the weight vector precisely when it is closest to the optimal direction.

The Projection and Rejection (PR) Product

To address this, the authors propose the Projection and Rejection (PR) Product. The core innovation lies in decoupling the gradient from the angle $\theta$ during the backward pass, ensuring a consistent signal for weight rotation.

Geometric comparison of standard inner product (a) and the PR Product (b) during backpropagation.

Mechanism

  • Forward Pass: The PR Product is identical to the standard inner product. It introduces no new parameters and maintains the same output for any given input and weight set.
  • Backward Pass: By rewriting the inner product and treating specific components as constants during differentiation (detaching them), the authors force the direction gradient to be independent of $\theta$. This provides a "hard" gradient that remains large even as vectors align, acting as a constant-pressure valve for optimization.

The authors implemented this logic across standard architectures, creating PR-FC, PR-CNN, and PR-LSTM layers.

Performance Evaluation

The PR Product was evaluated across image classification and image captioning benchmarks. In image classification tasks on CIFAR-10 and CIFAR-100, the PR Product consistently improved the performance of state-of-the-art architectures like ResNet, WideResNet, and DenseNet.

Error rates on CIFAR-10 and CIFAR-100 datasets across various architectures using standard vs. PR Product.

On the MS COCO image captioning dataset, a model utilizing PR-LSTM and PR-CNN achieved a CIDEr score of 128.3. This result outperformed several complex models that rely on additional semantic information like scene graphs, demonstrating that fundamental improvements to optimization can be more effective than architectural "bells and whistles."

Key Insights

  • Directional Importance: Ablation studies revealed that the direction gradient is significantly more critical for model accuracy than the length gradient. Removing the length gradient barely affected accuracy, while removing the direction gradient caused performance to collapse.
  • Feature Diversity: Analysis indicates that PR Product models tend to maintain higher orthogonality between weight and data vectors, potentially leading to more discriminative feature learning.
  • Efficiency: Since the forward pass is unchanged and the backward pass involves only minor additional operations, the PR Product offers a performance boost with virtually zero computational or memory overhead.

Conclusion

The PR Product serves as a robust, drop-in replacement for the standard inner product. By fixing the vanishing direction gradient problem, it enables more efficient and accurate training across a wide range of neural network architectures without increasing model complexity.

Summary updated

BibTeX
@inproceedings{wang2019pr,
  author = {Zhennan Wang and Wenbin Zou and Chen Xu},
  title = {{PR Product: A Substitute for Inner Product in Neural Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00611},
}
Figure