Abstract
Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly ($75\%$) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Quaternion attention Model and Quaternion Transformer. Extensive experiments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to $75\%$ reduction in parameter size without significant loss in performance.
Introduction
Modern Natural Language Processing (NLP) is dominated by massive neural architectures like the Transformer. While these models achieve stellar results, they are incredibly parameter-hungry, often containing millions or billions of parameters. This makes deployment on mobile devices or memory-constrained environments difficult.
This research introduces a novel way to shrink these massive models by up to 75% without sacrificing performance. By shifting computation from standard real-valued space to Quaternion space (hypercomplex numbers), the authors create "Quaternion Transformers" and "Quaternion Attention" mechanisms that are significantly more memory-efficient and, in some cases, even more accurate than their full-sized counterparts.
Methodology: The Power of Hypercomplex Space
The core innovation lies in replacing standard linear transformations with Quaternion algebra. A Quaternion ($Q$) is a hypercomplex number consisting of one real part and three imaginary parts ($i, j, k$).
The "secret sauce" of this approach is the Hamilton product ($\otimes$). When multiplying two Quaternions using the Hamilton product, the resulting operation creates a specific structure of weight sharing. In a standard neural network layer, a $4 \times 4$ transformation requires 16 unique parameters. In a Quaternion layer, the Hamilton product reduces this to just 4 unique parameters—a 75% reduction—while still allowing all four components of the input to interact with each other.
The authors developed several new components based on this math:
- Quaternion Feed-Forward Networks (QFFN): Replacing standard dense layers.
- Quaternion Attention: A new alignment mechanism where the "matching" between words happens in hypercomplex space.
- Quaternion Transformer: A complete overhaul of the Transformer architecture, offered in "partial" (only self-attention is quaternion) and "full" (all linear layers are quaternion) versions.
Experimental Evaluation
The models were tested across 13 different datasets to prove generalizability, including Natural Language Inference (SNLI, SciTail, MNLI), Question Answering (WikiQA), Paraphrase Detection (Quora), Sentiment Analysis (IMDb, SST), and Neural Machine Translation (IWSLT’15, WMT’14/16/18).
Key Results
- Parameter Savings: Across almost all tasks, the models achieved a 75% reduction in parameter count.
- Performance Gains: Surprisingly, the Quaternion models often outperformed larger models. On the Mathematical Language Understanding (MLU) task, the Quaternion Transformer saw an 8.3% accuracy gain over the standard Transformer while being 25% of the size.
- Sentiment and Translation: On the IMDb dataset, the "full" Quaternion Transformer improved accuracy by 1.3% with 75% fewer parameters. In English-to-Vietnamese translation, the "partial" version outperformed the base model by 2.5 BLEU points while saving 32% of parameters.
Discussion and Insights
The 75% reduction is a mathematical property of the Hamilton product. Beyond efficiency, the authors argue that Quaternions are a natural fit for language because words often have multiple meanings (senses). Representing a word as a four-component hypercomplex number aligns well with this multi-sense nature. Furthermore, unlike "multi-head" attention where heads are independent, Quaternion space explicitly encodes interactions between components, providing a stronger inductive bias for learning pairwise affinities.
Conclusion
This work demonstrates that hypercomplex representations are self-contained and interface seamlessly with standard real-valued layers. While challenges remain—such as finding the optimal initialization scheme for hypercomplex spaces—the results show that Quaternion-inspired models can significantly reduce memory footprints without significant loss in performance, and often with notable gains in accuracy and reasoning capabilities.
Summary updated