TernaryLLM: Ternarized Large Language Model

Tianqi Chen, Zhe Li, Weixiang Xu, Zeyu Zhu, Dong Li, Lu Tian, Emad Barsoum, Peisong Wang, Jian Cheng

Introduction

Large Language Models (LLMs) have revolutionized natural language processing but come with immense computational and memory requirements. While 4-bit and 8-bit quantization have become standard for deployment, pushing models to ultra-low bit widths—specifically ternary weights $\{-1, 0, 1\}$ (1.58 bits)—remains a significant challenge due to substantial accuracy degradation. TernaryLLM addresses this by introducing novel quantization and distillation techniques that allow 1.58-bit models to rival or even exceed the performance of 2-bit counterparts.

Challenges in LLM Ternarization

The transition from high-precision floating-point weights to ternary values faces three primary hurdles:

  1. Asymmetric Weight Distributions: LLM weights often exhibit non-zero means and asymmetric outliers, which standard symmetric ternary methods fail to capture accurately.
  2. Information Loss: Extreme quantization narrows the feature representation space and disrupts the semantic clustering learned during pre-training.
  3. Distillation Instability: Standard Knowledge Distillation (KD) using Mean Squared Error (MSE) is highly sensitive to the extreme activation outliers common in LLMs. These outliers dominate the loss function, leading to unstable training and "exploding" gradients.

Proposed Innovations

Dual Learnable Ternarization (DLT)

To handle asymmetric weight distributions, TernaryLLM employs Dual Learnable Ternarization (DLT). Unlike traditional methods that use a fixed symmetric threshold, DLT introduces two learnable parameters: a scale ($\alpha$) and a shift ($\gamma$). The weights are approximated as $W \approx \alpha \cdot T + \gamma$, where $T \in \{-1, 0, 1\}$. This added flexibility allows the quantizer to adapt to the specific distribution of each weight group, significantly reducing rounding errors compared to fixed-threshold methods.

Outlier-Friendly Feature Knowledge Distillation (OFF)

To recover lost representational power, the authors propose Outlier-Friendly Feature Knowledge Distillation (OFF). Instead of the standard MSE loss, which is vulnerable to activation outliers, OFF utilizes Cosine Similarity to align the student model's features with the teacher's.

Extracted figure from page 5 of TernaryLLM: Ternarized Large Language Model

As shown in Figure 5, MSE-based distillation (green line) suffers from massive loss spikes due to outliers, whereas the proposed cosine-based distillation (blue line) remains stable. By focusing on the direction of feature vectors rather than their absolute magnitude, OFF effectively maximizes the mutual information between the floating-point teacher and the ternary student.

Experimental Results

TernaryLLM was evaluated across several model families, including OPT and LLaMA. The results consistently show that the 1.58-bit TernaryLLM outperforms existing 2-bit quantization methods like GPTQ, AWQ, and OmniQuant.

Extracted table from page 7 of TernaryLLM: Ternarized Large Language Model

In the OPT model series (Figure 6), TernaryLLM achieves significantly lower perplexity on WikiText2 and C4 datasets compared to other low-bit methods. For instance, on OPT-6.7B, TernaryLLM reaches a perplexity of 13.81 on WikiText2, outperforming the 2-bit OmniQuant (14.43) while using fewer bits per weight.

On the LLaMA-3-8B model, TernaryLLM demonstrated even more impressive gains, outperforming the state-of-the-art 2-bit method by 5.8 points in perplexity on the C4 dataset and 8.2% in average zero-shot accuracy across benchmarks like PIQA and HellaSwag.

Conclusion

TernaryLLM proves that 1.58-bit quantization is a viable path for extreme LLM compression. By combining flexible learnable ternarization (DLT) with a robust, outlier-insensitive distillation strategy (OFF), the researchers have narrowed the gap between ultra-low-bit models and their full-precision counterparts. This work paves the way for deploying massive LLMs on resource-constrained hardware by replacing expensive multiplications with efficient additions.

BibTeX
@misc{chen2024ternaryllm,
  author = {Tianqi Chen and Zhe Li and Weixiang Xu and Zeyu Zhu and Dong Li and Lu Tian and Emad Barsoum and Peisong Wang and Jian Cheng},
  title = {{TernaryLLM: Ternarized Large Language Model}},
  year = {2024},
  eprint = {2406.07177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07177},
}
Figure