Abstract
Strictly enforcing orthonormality constraints on parameter matrices has been shown advantageous in deep learning. This amounts to Riemannian optimization on the Stiefel manifold, which, however, is computationally expensive. To address this challenge, we present two main contributions: (1) A new efficient retraction map based on an iterative Cayley transform for optimization updates, and (2) An implicit vector transport mechanism based on the combination of a projection of the momentum and the Cayley transform on the Stiefel manifold. We specify two new optimization algorithms: Cayley SGD with momentum, and Cayley ADAM on the Stiefel manifold. Convergence of Cayley SGD is theoretically analyzed. Our experiments for CNN training demonstrate that both algorithms: (a) Use less running time per iteration relative to existing approaches that enforce orthonormality of CNN parameters; and (b) Achieve faster convergence rates than the baseline SGD and ADAM algorithms without compromising the performance of the CNN. Cayley SGD and Cayley ADAM are also shown to reduce the training time for optimizing the unitary transition matrices in RNNs.
Introduction
In deep learning, enforcing orthonormality—ensuring that the columns of a weight matrix are unit length and mutually perpendicular—offers significant advantages. It helps prevent vanishing or exploding gradients in Recurrent Neural Networks (RNNs) and stabilizes activations in Convolutional Neural Networks (CNNs), often leading to superior generalization and faster convergence.
Mathematically, this task involves optimization on a curved surface called the Stiefel manifold. Standard Euclidean optimization methods like SGD are unsuitable because their update steps typically move parameters off the manifold, losing the orthonormality property. To maintain these constraints, researchers employ a "retraction map" to project updates back onto the manifold. However, traditional retractions, such as the closed-form Cayley transform or Singular Value Decomposition (SVD), require $O(n^3)$ matrix inversions or decompositions. These operations are computationally prohibitive for the large-scale parameter matrices found in modern neural networks.
Methodology
This work introduces two key innovations to make Riemannian optimization practical for deep learning:
1. Iterative Cayley Transform
Instead of computing the exact Cayley transform, which necessitates a costly matrix inverse, the authors propose an iterative fixed-point approach: $Y = X + \frac{\alpha}{2}W(X + Y)$. They demonstrate that as few as two iterations are sufficient to approximate the manifold with high precision (error $\approx 10^{-6}$). This replaces the $O(n^3)$ inverse with efficient, GPU-friendly matrix multiplications. Furthermore, this iterative approach is often more numerically stable than exact inversions in 32-bit float precision.
2. Implicit Vector Transport
Using momentum on a manifold is non-trivial because momentum vectors from previous steps reside in different "tangent spaces" than the current gradient. Standard practice requires explicit "vector transport" to move these vectors, adding significant overhead. The authors show that the Cayley transform implicitly handles this projection. By combining momentum and gradients in Euclidean space before applying the transform, they bypass explicit vector transport calculations entirely.
These innovations are realized in two new algorithms: Cayley SGD with momentum and Cayley ADAM.
Experimental Results
The algorithms were evaluated on standard benchmarks, including CIFAR-10/100 for CNNs (VGG and Wide ResNet) and the pixel-by-pixel MNIST task for RNNs.
Convergence and Accuracy
A primary finding is that while standard SGD and ADAM may be faster per individual iteration, the Cayley-based variants converge in far fewer epochs. The orthonormality constraint acts as a powerful regularizer and stabilizer.
As shown in Figure 3, Cayley SGD and Cayley ADAM exhibit much faster initial convergence compared to their Euclidean counterparts and other orthogonal methods like SRIP. On a Wide ResNet (WRN 28-10), Cayley ADAM achieved a top error rate of 3.57% on CIFAR-10, outperforming standard ADAM (3.86%) and SGD (3.89%).
Computational Efficiency
The iterative approach provides a massive speedup over exact methods. For a WRN 28-10, Cayley SGD required approximately 218 seconds per epoch, whereas the closed-form Cayley transform took 1071 seconds—a nearly 5x improvement. In RNN tasks, the proposed methods reduced training time per iteration by approximately 35% compared to existing state-of-the-art "hard" orthogonal methods.
Figure 4 further illustrates the stability and consistent loss reduction across different datasets and iteration counts, highlighting the robustness of the iterative Cayley approach.
Conclusion
By replacing expensive matrix operations with an iterative approximation and simplifying momentum handling, this work makes "hard" orthonormality constraints viable for large-scale deep learning. The resulting Cayley SGD and Cayley ADAM algorithms offer a compelling combination of theoretical grounding, computational efficiency, and improved empirical performance. Future work may explore the application of these methods to newer architectures like Transformers, where orthogonal attention heads could provide similar benefits.
Summary updated