Abstract
Contemporary state-of-the-art neural networks have increasingly large numbers of parameters, which prevents their deployment on devices with limited computational power. Pruning is one technique to remove unnecessary weights and reduce resource requirements for training and inference. In addition, for ML tasks where the input data is multi-dimensional, using higher-dimensional data embeddings such as complex numbers or quaternions has been shown to reduce the parameter count while maintaining accuracy. In this work, we conduct pruning on real and quaternion-valued implementations of different architectures on classification tasks. We find that for some architectures, at very high sparsity levels, quaternion models provide higher accuracies than their real counterparts. For example, at the task of image classification on CIFAR-10 using Conv-4, at 3% of the number of parameters as the original model, the pruned quaternion version outperforms the pruned real by more than 10%. Experiments on various network architectures and datasets show that for deployment in extremely resource-constrained environments, a sparse quaternion network might be a better candidate than a real sparse model of similar architecture.
Introduction
Modern deep learning models are massive, often containing hundreds of millions of parameters. This scale makes them nearly impossible to deploy on "edge" devices like mobile phones or IoT sensors due to limited memory and computational power. Two primary strategies have emerged to address this: weight pruning, which removes redundant connections from standard real-valued networks, and higher-dimensional embeddings like quaternions, which reduce parameter counts through weight sharing. This study investigates the synergy between these two approaches, specifically examining if quaternion networks maintain accuracy better than real networks under extreme pruning.
Methodology
The study compares standard Real-valued networks (R) against Quaternion-valued networks (Q).
Quaternion-Valued Layers
Unlike real numbers, quaternions (4D numbers) utilize the Hamilton product for multiplication. This mathematical structure allows a single quaternion weight to perform the work of four real weights, effectively reducing the parameter count by 75% while maintaining the same number of neurons. For color images, RGB channels are mapped to the three imaginary components, with the grayscale value occupying the fourth component.
Pruning and Training
The authors employed Iterative Global Pruning, a magnitude-based approach that removes the smallest 20% of weights in each iteration. They also tested the Lottery Ticket Hypothesis (LTH) to determine if "winning tickets" exist for quaternion models. To ensure a fair comparison, architectures were designed such that real and quaternion versions possessed an equivalent number of real-valued neurons.
Experimental Results
The models were evaluated on MNIST, CIFAR-10, and CIFAR-100 datasets.
The Sparsity Crossover
A key finding is the "sparsity crossover." While real-valued networks often exhibit higher baseline accuracy at low sparsity levels, their performance tends to collapse as pruning becomes extreme. In contrast, quaternion networks are more robust, with accuracy dropping gracefully. At a critical point of extreme sparsity (e.g., 3% of original parameters for Conv-4 on CIFAR-10), the quaternion model significantly outperforms the real model, sometimes by over 10%.
Efficiency and Convergence
The experiments confirmed that the Lottery Ticket Hypothesis applies to quaternions, allowing for the identification of efficient sparse structures early in training. While real models often begin to fail at 10% sparsity, quaternion models remain competitive even as they approach 1-2% of their original parameter count.
Conclusion and Future Work
Sparse quaternion networks offer a promising path for ultra-lightweight model deployment. However, challenges remain. The quaternion advantage was less pronounced in deeper architectures like ResNet-18 and VGG-16, possibly due to Batch Normalization complexities or network depth. Furthermore, the interaction between quaternions and early stopping requires deeper investigation, as real networks currently show better performance under early termination. Future work will explore these benefits in more complex tasks such as semantic segmentation and speech recognition.
Summary updated