Kronecker Decomposition for GPT Compression

Ali Edalati, Marzieh S. Tahaei, Ahmad Rashid, Vahid Partovi Nia, James J. Clark, Mehdi Rezagholizadeh

Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Nia, James Clark, Mehdi Rezagholizadeh. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022.

Introduction

Generative Pre-trained Transformers (GPT) are the heavyweights of the NLP world, but their massive parameter counts make them prohibitive for deployment on edge devices like smartphones or IoT hardware. While encoder-based models (like BERT) have seen a flurry of successful compression research, auto-regressive models (like GPT) have lagged behind. The existing baseline, DistilGPT2, requires massive computational resources—training for three full epochs on the massive OpenWebText dataset.

Researchers have introduced KnGPT2, a compressed version of GPT-2 that uses Kronecker decomposition to slash model size while maintaining high performance. By replacing large weight matrices with smaller Kronecker factors and using a lightweight distillation process, KnGPT2 outperforms DistilGPT2 on both language modeling and NLU tasks, despite being trained on 10x less data and in a fraction of the time.

Methodology

The core innovation is the application of Kronecker Decomposition to the linear layers of GPT-2. Instead of storing a large weight matrix $W$, the authors represent it as the Kronecker product of two much smaller matrices ($W \approx A \otimes B$). This mathematically elegant trick turns a matrix of size $(m_1m_2) \times (n_1n_2)$ into two matrices of size $m_1 \times n_1$ and $m_2 \times n_2$, significantly reducing the parameter count.

To recover the performance lost during decomposition, the model undergoes a "light" pre-training phase called Intermediate Layer Knowledge Distillation (ILKD). It learns from a "teacher" (the original GPT-2) by matching attention distributions using Kullback–Leibler (KL) divergence and hidden states using Mean Squared Error (MSE) across all layers. Unlike DistilGPT2, which is trained for 3 epochs on the full OpenWebText dataset, KnGPT2 is pre-trained on only 10% of the data for just 1 epoch.

Key Results and Efficiency

KnGPT2 demonstrates superior performance across both language modeling and natural language understanding (NLU) benchmarks. On the GLUE benchmark, KnGPT2 achieved an average score of 77.42, significantly beating DistilGPT2's 74.52 and coming remarkably close to the original full-sized GPT-2 Small (77.56).

Performance comparison on the GLUE benchmark across different models.

Beyond raw performance, the efficiency gains are the most striking aspect of this approach. KnGPT2's training pipeline is roughly 14x faster than DistilGPT2 (6.5 hours vs. >90 hours) because it requires far less data to reach convergence. It achieved these results using only 3.2 GB of training data, compared to the 38 GB used by the baseline.

Comparison of parameter counts, training time, and dataset requirements.

Conclusion

KnGPT2 proves that Kronecker decomposition is particularly effective for large language models because it preserves the block-like structural relationships within weight matrices better than simple pruning. The results suggest that with the right compression and distillation strategy, GPT models can become much more competitive in NLU tasks, a domain where BERT-style models typically dominate. Future work will involve testing this method on even larger models like GPT-3 and exploring higher compression factors to identify the limits of this technique.

Summary updated

BibTeX
@inproceedings{edalati2022kronecker,
  author = {Ali Edalati and Marzieh S. Tahaei and Ahmad Rashid and Vahid Partovi Nia and James J. Clark and Mehdi Rezagholizadeh},
  title = {{Kronecker Decomposition for GPT Compression}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=CvmuB5gBwaw},
}
Figure