Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Nia, James Clark, Mehdi Rezagholizadeh. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022.
Introduction
Generative Pre-trained Transformers (GPT) are the heavyweights of the NLP world, but their massive parameter counts make them prohibitive for deployment on edge devices like smartphones or IoT hardware. While encoder-based models (like BERT) have seen a flurry of successful compression research, auto-regressive models (like GPT) have lagged behind. The existing baseline, DistilGPT2, requires massive computational resources—training for three full epochs on the massive OpenWebText dataset.
Researchers have introduced KnGPT2, a compressed version of GPT-2 that uses Kronecker decomposition to slash model size while maintaining high performance. By replacing large weight matrices with smaller Kronecker factors and using a lightweight distillation process, KnGPT2 outperforms DistilGPT2 on both language modeling and NLU tasks, despite being trained on 10x less data and in a fraction of the time.
Methodology
The core innovation is the application of Kronecker Decomposition to the linear layers of GPT-2. Instead of storing a large weight matrix $W$, the authors represent it as the Kronecker product of two much smaller matrices ($W \approx A \otimes B$). This mathematically elegant trick turns a matrix of size $(m_1m_2) \times (n_1n_2)$ into two matrices of size $m_1 \times n_1$ and $m_2 \times n_2$, significantly reducing the parameter count.
To recover the performance lost during decomposition, the model undergoes a "light" pre-training phase called Intermediate Layer Knowledge Distillation (ILKD). It learns from a "teacher" (the original GPT-2) by matching attention distributions using Kullback–Leibler (KL) divergence and hidden states using Mean Squared Error (MSE) across all layers. Unlike DistilGPT2, which is trained for 3 epochs on the full OpenWebText dataset, KnGPT2 is pre-trained on only 10% of the data for just 1 epoch.
Key Results and Efficiency
KnGPT2 demonstrates superior performance across both language modeling and natural language understanding (NLU) benchmarks. On the GLUE benchmark, KnGPT2 achieved an average score of 77.42, significantly beating DistilGPT2's 74.52 and coming remarkably close to the original full-sized GPT-2 Small (77.56).
Beyond raw performance, the efficiency gains are the most striking aspect of this approach. KnGPT2's training pipeline is roughly 14x faster than DistilGPT2 (6.5 hours vs. >90 hours) because it requires far less data to reach convergence. It achieved these results using only 3.2 GB of training data, compared to the 38 GB used by the baseline.
Conclusion
KnGPT2 proves that Kronecker decomposition is particularly effective for large language models because it preserves the block-like structural relationships within weight matrices better than simple pruning. The results suggest that with the right compression and distillation strategy, GPT models can become much more competitive in NLU tasks, a domain where BERT-style models typically dominate. Future work will involve testing this method on even larger models like GPT-3 and exploring higher compression factors to identify the limits of this technique.
Summary updated