Abstract
We present new architectures for feedforward neural networks built from products of learned or random low-dimensional rotations that offer substantial space compression and computational speedups. KAMA-NNs (Kac's Asymptotic Matrix Approximator) leverage Givens rotations to maintain orthogonality, providing a structured approach that outperforms traditional dense layers and other structured alternatives in kernel estimation and reinforcement learning tasks.
Introduction
Standard deep learning models rely on dense weight matrices, which are parameter-heavy and computationally expensive. This poses challenges for deployment on resource-constrained devices. While structured matrices like Toeplitz or Hadamard transforms offer compression, they often lack orthogonality—a property crucial for training stability and the accuracy of kernel-based estimators. KAMA-NNs address this by constructing weight matrices from products of 2D rotations, ensuring built-in orthogonality and high efficiency.
Methodology
The fundamental unit of a KAMA-NN is the Givens rotation, which transforms a vector within a 2D plane of a higher-dimensional space. A KAMA layer represents a $d \times d$ weight matrix as a product of $k$ such rotations. This decomposition allows for $O(1)$ time per rotation; with $k = O(d \log d)$, the total complexity is significantly reduced compared to the $O(d^2)$ of dense layers.
In adaptive settings, the angles of these rotations are learned. The authors employ manifold optimization, specifically coordinate gradient ascent on the $SO(d)$ manifold. This geometric approach ensures the weights remain orthogonal throughout training. Theoretically, random KAMA-NNs are shown to be more accurate estimators for Pointwise Nonlinear Gaussian (PNG) kernels than standard unstructured networks.
Experimental Results
The performance of KAMA-NNs was evaluated in two regimes: non-adaptive kernel estimation and adaptive reinforcement learning.
Kernel Estimation
In kernel estimation tasks across datasets like boston, wine, and g50, KAMA-NNs consistently demonstrated lower Mean Squared Error (MSE) than unstructured Gaussian baselines (URN). They also matched or exceeded the performance of other state-of-the-art orthogonal estimators.
Reinforcement Learning
In reinforcement learning, KAMA-NNs were used to parameterize policies for MuJoCo locomotion tasks. The architecture achieved up to 7x parameter reduction compared to fully connected baselines without sacrificing performance. Notably, KAMA-NNs outperformed other structured baselines, such as Toeplitz-based networks, in terms of cumulative reward.
Discussion and Conclusion
KAMA-NNs provide a powerful framework for model compression through low-dimensional rotations. By maintaining orthogonality and leveraging logarithmic scaling, they offer a superior trade-off between efficiency and accuracy. While the sequential nature of Givens rotations presents a parallelization challenge for current GPUs, the theoretical gains and empirical performance on edge-relevant tasks are significant. Future work includes extending this architecture to Transformers and deep CNNs, as well as exploring learned subspace selection for rotations.
Summary updated