Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

Yi Tay, Aston Zhang, Anh Tuan Luu, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, Siu Cheung Hui

Abstract

Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly (75%) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Quaternion attention Model and Quaternion Transformer. Extensive experiments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to 75% reduction in parameter size without significant loss in performance.

Introduction

The rapid advancement of Natural Language Processing (NLP) has been driven by increasingly large and complex neural architectures, most notably the Transformer. However, these models are notoriously "parameter-hungry," often requiring millions or billions of weights. This high parameter count leads to significant memory overhead, making deployment challenging on edge devices or in resource-constrained environments.

While common compression techniques like pruning or quantization offer some relief, they often act as post-hoc fixes rather than fundamental improvements. This research proposes a more foundational shift: moving neural computation from real-valued space ($\mathbb{R}$) to hypercomplex Quaternion space ($\mathbb{H}$). By leveraging the unique mathematical properties of Quaternions, specifically the Hamilton product, we can introduce a strong architectural inductive bias that maintains expressive power while drastically reducing the model's footprint.

Methodology: Quaternion Networks and the Hamilton Product

A Quaternion is a hypercomplex number defined as $Q = r + xi + yj + zk$, where $r$ is the real part and $x, y, z$ are imaginary parts. In a Quaternion neural network, each "weight" or "feature" is represented as such a 4-dimensional entity.

The core mechanism for parameter efficiency is the Hamilton product ($\otimes$), which defines how two Quaternions are multiplied. In a standard linear layer, a transformation between 4-dimensional vectors would typically require a $4 \times 4$ matrix of 16 independent weights. In contrast, a Quaternion linear layer using the Hamilton product is governed by only 4 unique parameters ($W_r, W_x, W_y, W_z$).

The Hamilton product structure enables a 75% reduction in parameters by sharing weights across the four quaternion components.

As shown in the figure above, the output components ($r', x', y', z'$) are computed through a specific pattern of interactions between the input components ($r, x, y, z$) and the shared weight variables. This structure mathematically enforces a 75% reduction in the degrees of freedom compared to a standard real-valued layer. Beyond efficiency, the Hamilton product forces the four components to interact in a complex, predefined way, which serves as a powerful inductive bias for capturing the "multi-sense" nature of language.

Proposed Architectures

We introduce two primary architectures:

  1. Quaternion Attention (Q-Att): This replaces standard dot-product attention with a mechanism based on the Hamilton product for alignment.
  2. Quaternion Transformer: A comprehensive adaptation of the Transformer where linear transformations (Query, Key, Value) and Feed-Forward Networks (FFNs) are replaced with their Quaternion counterparts. We evaluated both "full" (all layers converted) and "partial" (only self-attention converted) variants.

Experimental Evaluation

The proposed models were evaluated across a diverse battery of 13 NLP datasets, covering tasks such as Natural Language Inference (SNLI, MNLI), Question Answering (WikiQA), Sentiment Analysis (IMDb, SST), and Machine Translation (IWSLT, WMT).

Key Results

  • Compression and Performance: Across nearly all tasks, the "full" Quaternion models achieved a 75% reduction in parameters compared to vanilla real-valued models. Remarkably, this compression often came with performance improvements. For instance, on the IMDb sentiment analysis task, the Quaternion Transformer (full) achieved 83.9% accuracy, outperforming the vanilla Transformer (82.6%) despite being four times smaller.
  • Reasoning and Translation: On the Mathematical Language Understanding (MLU) task, the "partial" Quaternion Transformer outperformed the base model by +8.3%, suggesting that hypercomplex representations are particularly adept at capturing structured, symbolic relationships. In Machine Translation (En-Vi), the partial model achieved a +2.5 BLEU gain while saving 32% of parameters.

Discussion and Conclusion

The success of Quaternion Networks suggests that smarter parameterization can be more effective than brute-force scaling. The 4-component structure of Quaternions acts as a built-in "multi-head" mechanism, where components are explicitly encouraged to interact. This allows the model to capture different facets of a word's meaning within its imaginary components.

While promising, challenges remain. Current deep learning hardware and libraries are optimized for real-valued matrix operations, making the efficient implementation of Hamilton products a practical hurdle. Furthermore, the sensitivity of these models to initialization schemes suggests that the optimization landscape in hypercomplex space requires further study. Future work may explore even higher-dimensional spaces, such as Octonions, which could theoretically offer an 87.5% reduction in parameters.

In conclusion, Quaternion Networks provide a robust framework for building lightweight, memory-efficient NLP models without sacrificing the expressive power necessary for complex linguistic tasks.

Summary updated

Abstract

Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly (75%) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Quaternion attention Model and Quaternion Transformer. Extensive experiments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to 75% reduction in parameter size without significant loss in performance.
BibTeX
@inproceedings{tay2019lightweight,
  author = {Yi Tay and Aston Zhang and Anh Tuan Luu and Jinfeng Rao and Shuai Zhang and Shuohang Wang and Jie Fu and Siu Cheung Hui},
  title = {{Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=krhuwi6nDda},
}
Figure