LAMP: Large Model Pruning with Inter-Block Error Compensation

Xin Luo, Zihang Jiang, S Kevin ZhouOpenReview 2025 · Submission to ICLR 2025 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

The increasing prevalence of large-scale models, both in vision and language domains, presents significant challenges in terms of memory and resource consumption. While model pruning is an effective method for compressing models to alleviate these constraints, existing techniques either require extensive fine-tuning, which is resource-intensive, or perform well only at low sparsity levels (10%-50%), failing at high sparsity levels (50%-90%). To address these issues, this paper introduces LAMP to mitigate the drawbacks associated with traditional pruning methods, namely high resource consumption in methods that require extensive fine-tuning, and poor performance at high sparsity levels in methods that do not. It reduces memory overhead and alleviates performance degradation at high sparsity. Experimental results demonstrate that LAMP achieves slightly better performance than SparseGPT at low sparsity levels and significantly better at high sparsity levels in both language and vision models, without significantly increasing memory consumption when compared to SparseGPT.
BibTeX
@misc{luo2025lamp,
  author = {Xin Luo and Zihang Jiang and S Kevin Zhou},
  title = {{LAMP: Large Model Pruning with Inter-Block Error Compensation}},
  year = {2025},
  url = {https://openreview.net/forum?id=mclaeTduHp},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}

Loading related content...

Figure