Abstract
GPT is an auto-regressive Transformer-based pre-trained language model which has attracted a lot of attention in the natural language processing (NLP) domain due to its state-of-the-art performance in several downstream tasks. The success of GPT is mostly attributed to its pre-training on huge amount of data and its large number of parameters (from ~100M to billions of parameters). Despite the superior performance of GPT (especially in few-shot or zero-shot setup), this overparameterized nature of GPT can be very prohibitive for deploying this model on devices with limited computational power or memory. This problem can be mitigated using model compression techniques; however, compressing GPT models has not been investigated much in the literature. In this work, we use Kronecker decomposition to compress the linear mappings of the GPT-2 model. Our Kronecker GPT-2 model (KnGPT2) is initialized based on the Kronecker decomposed version of the GPT-2 model and then is undergone a very light pre-training on only a small portion of the training data with intermediate layer knowledge distillation (ILKD). Finally, our KnGPT2 is fine-tuned on down-stream tasks using ILKD as well. We evaluate our model on both language modeling and General Language Understanding Evaluation benchmark tasks and show that with more efficient pre-training and similar number of parameters, our KnGPT2 outperforms the existing DistilGPT2 model significantly.
Introduction
Large-scale Generative Pre-trained Transformers (GPT) have revolutionized NLP, but their massive size—often reaching billions of parameters—presents a significant barrier for deployment on edge devices with limited memory and compute. While the BERT family has benefited from numerous compression techniques like DistilBERT, the auto-regressive GPT family has fewer efficient alternatives. The current standard, DistilGPT2, requires intensive pre-training on massive datasets. This paper introduces KnGPT2, a compressed variant that utilizes Kronecker decomposition to achieve superior performance with a fraction of the training resources.
Methodology
The core innovation of KnGPT2 lies in applying Kronecker Decomposition to the linear layers of the GPT-2 architecture. Instead of maintaining a standard large weight matrix $W$, it is represented as the Kronecker product of two smaller matrices ($W \approx A \otimes B$). This structural preservation allows the model to maintain the same number of layers and output dimensions as the original teacher model.
Initialization and Distillation
To ensure a high-quality starting point, the factors $A$ and $B$ are initialized using a rank-1 Singular Value Decomposition (SVD) of the reshaped original weight matrix. Following initialization, the model undergoes Intermediate Layer Knowledge Distillation (ILKD). During this phase, KnGPT2 learns from a full GPT-2 teacher by minimizing the difference in embeddings, attention distributions (via KL Divergence), and hidden states (via MSE). The authors found that decomposing only the odd-numbered transformer layers provided the optimal balance between parameter reduction and accuracy.
Experimental Results
KnGPT2 was evaluated against DistilGPT2 and the original GPT-2 Small across several benchmarks. The results demonstrate that Kronecker decomposition preserves the "knowledge" of the teacher model more effectively than architectural reduction (removing layers) alone.
Training Efficiency
One of the most striking findings is the efficiency of the training process. KnGPT2 was pre-trained for only one epoch on a 10% subset (3.2 GB) of the OpenWebText dataset. As shown in the comparison below, this resulted in a nearly 14x reduction in training time compared to DistilGPT2.
Benchmark Performance
Despite the significantly reduced training budget, KnGPT2 outperformed DistilGPT2 in both language modeling and downstream tasks. On the WikiText-103 dataset, KnGPT2 achieved a test perplexity of 20.5, compared to 23.7 for DistilGPT2. Furthermore, on the GLUE benchmark test set, KnGPT2 + ILKD achieved an average score of 79.25, closely approaching the performance of the original GPT-2 Small.
Conclusion
KnGPT2 demonstrates that Kronecker decomposition is a highly effective tool for compressing large language models. By preserving the structural integrity of the original matrices and utilizing targeted distillation, KnGPT2 achieves better accuracy than existing baselines while requiring 10x less data and 14x less training time. Future work will explore scaling this method to even larger models like GPT-3 and testing its performance on complex generative tasks.
Summary updated