Exploring Extreme Quantization in Spiking Language Models

Malyaban Bal, Yi Jiang, Abhronil Sengupta

Introduction

As Large Language Models (LLMs) continue to scale, their energy consumption has become a critical bottleneck for sustainable AI development. While the human brain operates with remarkable efficiency using sparse, asynchronous signals, modern deep learning architectures typically rely on power-intensive, high-precision floating-point operations. Spiking Neural Networks (SNNs) offer a promising bio-plausible alternative by utilizing discrete binary "spikes" for communication. However, most existing SNNs still utilize 32-bit weights, limiting their potential for extreme hardware optimization.

This paper introduces SpikingBERT, the first 1-bit and 1.58-bit (ternary) Spiking Language Model. By integrating extreme weight quantization with a novel training framework, the researchers demonstrate that it is possible to achieve performance competitive with full-precision models while enabling a projected 9x increase in energy efficiency.

SpikingBERT Architecture and Methodology

The SpikingBERT framework is built upon three fundamental technical pillars designed to overcome the challenges of training discrete, low-precision networks.

1. Extreme Weight Quantization

The researchers implemented two primary variants of the model: a 1-bit version (binary weights: +1, -1) and a 1.58-bit version (ternary weights: -1, 0, 1). This approach replaces traditional floating-point multiplications with simple additions or bitwise operations, significantly reducing the memory footprint and computational complexity of the encoder-based BERT architecture.

2. Implicit Differentiation at Equilibrium

Traditional training for SNNs often relies on Backpropagation Through Time (BPTT), which is notoriously memory-intensive as it requires storing every intermediate state across the temporal dimension. To mitigate this, SpikingBERT utilizes implicit differentiation. By focusing on the "steady-state" or equilibrium of the network's spiking rate, the authors eliminated the need for surrogate gradients and drastically reduced the memory overhead during training.

3. Internal-Layer Knowledge Distillation (KD)

To bridge the accuracy gap introduced by extreme quantization, a "Teacher-Student" training regime was employed. A full-precision, non-spiking BERT model acts as the teacher, guiding the quantized spiking student. Specifically, the training process aligns the Average Spiking Rate (ASR) of the student’s layers with the outputs of the teacher’s corresponding layers. The authors noted that without this internal-layer KD, quantization resulted in a substantial 6–8% performance degradation.

Experimental Results

The model was rigorously evaluated using the General Language Understanding Evaluation (GLUE) benchmark, including tasks such as QQP, MNLI, SST-2, QNLI, and MRPC.

Accuracy and Efficiency

The 1.58-bit SpikingBERT achieved remarkable results, reaching 87.1% accuracy on SST-2 and 85.4% on QQP. These figures trail the full-precision spiking model by only 1-2%, while outperforming other efficient implementations like BinaryBERT and TernaryBERT across nearly all tasks.

Beyond accuracy, the theoretical energy gains are significant. When implemented on specialized 45nm CMOS hardware, the 1.58-bit model is estimated to offer a 9x energy efficiency improvement over standard floating-point models. This efficiency stems from the architecture's ability to replace power-hungry multiplications with energy-efficient integer accumulations.

The Temporal Advantage

A key insight from the study is the role of time steps ($T$) in SNNs. Even with extremely low-precision weights and activations, the model can "spread" information across the temporal dimension. This temporal processing effectively recovers the representational precision lost during the quantization process, allowing the model to maintain high performance despite its 1-bit or 1.58-bit constraints.

Conclusion and Future Directions

SpikingBERT represents a significant milestone in the development of energy-efficient language models. By proving that extreme quantization is viable for complex NLP tasks, this research opens new avenues for deploying LLMs on edge devices and neuromorphic hardware.

While the current study focuses on encoder-based models for classification, future work will target decoder-based generative models (such as GPT) and aim to close the remaining 1-2% accuracy gap through more advanced training regimes. The ultimate realization of these energy savings will depend on the continued maturation of specialized BNN and neuromorphic accelerators.

Summary updated

BibTeX
@inproceedings{bal2024exploring,
  author = {Malyaban Bal and Yi Jiang and Abhronil Sengupta},
  title = {{Exploring Extreme Quantization in Spiking Language Models}},
  booktitle = {ICONS},
  year = {2024},
  url = {https://openreview.net/forum?id=62D3T6tycV},
}
Figure