LEMON: Reviving Stronger and Smaller LMs from Larger LMs with Linear Parameter Fusion

Yilong Chen, Junyuan Shang, Zhenyu Zhang, Shiyao Cui, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu

Introduction

As Large Language Models (LLMs) continue to scale, there is a growing demand for Small Language Models (SLMs) that are efficient for edge deployment and low-latency applications. Traditionally, SLMs are either trained from scratch or derived via structured pruning. However, training from scratch often hits a performance ceiling defined by scaling laws, while pruning—which involves discarding "unimportant" neurons or layers—can lead to significant information loss.

LEMON (Linear Parameter Fusion) addresses these challenges by proposing a "fusion" approach instead of the traditional "search and destroy" pruning method. Rather than treating parameters as binary (keep or discard), LEMON learns to compress a large model into a smaller one by fusing parameters, thereby preserving functional knowledge more effectively from the entire source model.

Methodology: From Pruning to Fusion

The core innovation of LEMON is replacing binary pruning with learnable linear operators that transform large parameter matrices into smaller ones. This ensures that the resulting SLM inherits a distilled version of the larger model's entire weight distribution.

Comparison of pruning-based and fusion-based methods for model compression.

Linear Parameter Fusion

LEMON decomposes the fusion process into two primary operations:

  1. Vertical Fusion (Layer Operator): LLMs often exhibit modularity where adjacent layers perform similar functions. LEMON learns a mapping matrix ($\gamma$) to combine multiple layers from the source model into a single target layer.
  2. Horizontal Fusion (Dimension Operator): To reduce hidden dimensions (e.g., shrinking a 4096-dimension vector to 2560) without a parameter explosion, LEMON utilizes Kronecker products to create efficient "Dimension Operators."
  3. Controllable Receptive Fields: A "granularity" parameter ensures structural integrity, such as maintaining the distinctness of attention heads during the fusion process.

The framework operates in two stages: an initial phase where fusion operators are learned using a minimal dataset (0.26B tokens), followed by standard continued pre-training of the resulting smaller model.

Empirical Motivation: Functional Clusters

A critical observation justifying the fusion approach is the discovery of functional clusters within LLMs. Using Centered Kernel Alignment (CKA), the authors found that layers, particularly in the middle and deeper sections of the model, exhibit high similarity.

Similarity matrix and curve showing functional clusters of layers in LLMs, justifying the fusion of adjacent layers.

This high similarity suggests that layers are not unique entities but part of functional groups, making them ideal candidates for fusion rather than arbitrary deletion. By fusing these clusters, the small model inherits the collective knowledge of the group.

Evaluation and Results

LEMON was evaluated using LLaMA 2-7B as a source to produce 1.3B and 2.7B models. Testing across 11 benchmarks, including MMLU, HellaSwag, and ARC, revealed several key findings:

  • Superior Performance: LEMON-2.7B achieved an average score of 57.6 across benchmarks, outperforming the pruning-based Sheared-LLaMA-2.7B (56.7) and the scratch-trained INCITE-Base-3B (54.7).
  • High Efficiency: The initialization budget for LEMON is remarkably low. LEMON-2.7B reached the perplexity of state-of-the-art pruning baselines in 62.5% fewer steps, representing a significant saving in training resources.
  • Resource Savings: Compared to models trained from scratch like Pythia, LEMON required only 5.26% of the total pre-training tokens to achieve superior results.

Conclusion and Limitations

LEMON demonstrates that "reviving" a smaller model through parameter fusion is significantly more effective than traditional pruning. By treating all parameters as valuable and learning how to combine them, LEMON provides a high-quality starting point for SLM training, drastically reducing the required computational budget.

While effective, the current study is limited to linear fusion; non-linear methods might capture more complex relationships but would increase optimization difficulty. Additionally, while the final model is small, the fusion learning phase still requires significant memory to keep the large source model in the loop. Future work will explore scaling these patterns to models beyond the 7B parameter range.

Summary updated

Abstract

In the new era of language models, small models (with billions of parameter sizes) are receiving increasing attention due to their flexibility and cost-effectiveness in deployment. However, limited by the model size, the performance of small models trained from scratch may often be unsatisfactory. Learning a stronger and smaller model with the help of larger models is an intuitive idea. Inspired by the observing modular structures in preliminary analysis, we propose LEMON to learn competent initial points for smaller models by fusing parameters from larger models, thereby laying a solid foundation for subsequent training. Specifically, the parameter fusion process involves two operators for layer and dimension, respectively, and we also introduce controllable receptive fields to model the prior parameter characteristics. In this way, the larger model could be transformed into any specific smaller scale and architecture. Starting from LLaMA 2-7B, we revive two stronger and smaller models with 1.3B and 2.7B. Experimental results demonstrate that the fusion-based method exhibits flexibility and outperforms a series of competitive baselines in terms of both effectiveness and efficiency.
BibTeX
@inproceedings{chen2024lemon,
  author = {Yilong Chen and Junyuan Shang and Zhenyu Zhang and Shiyao Cui and Tingwen Liu and Shuohuan Wang and Yu Sun and Hua Wu},
  title = {{LEMON: Reviving Stronger and Smaller LMs from Larger LMs with Linear Parameter Fusion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.434},
}
Figure