Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models

Michele Mastromattei, Fabio Massimo ZanzottoACL 2024 · DOI

Abstract

As Large Language Models (LLMs) continue to scale, the computational and storage costs for deployment have become prohibitive. We present KEN (Kernel density Estimator for Neural network compression), a simple, non-parametric pruning algorithm that identifies critical parameters in fine-tuned LLMs by analyzing weight distributions. By retaining only the most statistically significant weight updates and resetting the remainder to their original pre-trained values, KEN reduces model size by 25% to 70%. Experimental results demonstrate that KEN-pruned models often match or exceed the performance of their unpruned counterparts across various benchmarks.

Introduction

The deployment of LLMs is often hindered by the "storage problem": even when models are pruned, they frequently require the full matrix structure for loading, yielding no actual disk space savings. Existing pruning methods are often computationally expensive, requiring gradient calculations, or are limited to specific architectures.

KEN addresses these challenges by offering an unstructured pruning method that is computationally light and results in a truly smaller file size. Inspired by the Winning Ticket Hypothesis, KEN seeks to identify the essential subnetworks within a large model that drive performance on specific tasks.

Methodology: The KEN Algorithm

The core innovation of KEN is using statistical distributions as a proxy for parameter importance. The algorithm operates in three primary steps:

  1. KDE Calculation: For every row in a fine-tuned weight matrix, KEN calculates a Kernel Density Estimation (KDE), creating a smoothed map of the weight distribution.
  2. Parameter Selection: It identifies the $k$ parameters that best fit this distribution (those with the highest likelihood). These are deemed the "essential" updates learned during fine-tuning.
  3. Value Reset: Instead of zeroing out unimportant weights—which can be destructive—KEN resets them to their original pre-trained values.
The KEN workflow: from fine-tuned model to optimized matrix via KDE parameter selection and pre-trained value resetting.

This process results in an optimized matrix where only a fraction of weights differ from the base model. To save space, only these $k$ modified weights and a small support dictionary mapping their positions need to be stored.

Experimental Evaluation

The authors evaluated KEN across seven model architectures (including BLOOM, BERT, and DeBERTa) and a diverse range of text classification tasks such as AG-NEWS, IMDB, and GLUE-SST2.

Performance and Compression

In nearly all test cases, KEN-pruned models achieved F1-scores equal to or higher than the original unpruned models. The algorithm successfully reset between 25% and 87% of parameters depending on the task. For example, a BLOOM-1B7 model maintained its performance on AG-NEWS even after resetting 74% of its fine-tuned parameters, reducing storage from ~7 GB to ~2.1 GB.

Impact of the $k$ Parameter

The variable $k$ serves as the primary control for the compression rate. By adjusting $k$, users can balance the trade-off between model size and performance retention.

Comparison of low and high k values on weight distribution and parameter selection.

Key Insights

  • Resetting > Masking: Reverting non-essential weights to their pre-trained state rather than zero allows the model to retain foundational knowledge, preventing the pruning process from "breaking" reasoning capabilities.
  • Uniformity: Visualization via $KEN_{viz}$ shows that KEN selects parameters uniformly across matrices, maintaining the structural integrity of transformer layers.
  • Efficiency: KEN consistently outperformed benchmarks like FLOP and BMP, and proved more parameter-efficient than LoRA (Low-Rank Adaptation) in several scenarios.

Conclusion

KEN provides a simple yet powerful framework for LLM compression. By leveraging non-parametric statistical tools, it achieves significant storage savings without the need for complex hyperparameter tuning or expensive gradient passes. While currently focused on sequence classification, future work will explore its application in generative tasks and further optimize the computational overhead of KDE calculations.

Summary updated

BibTeX
@inproceedings{mastromattei2024less,
  author = {Michele Mastromattei and Fabio Massimo Zanzotto},
  title = {{Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.674},
}
Figure