Introduction
As Large Language Models (LLMs) continue to scale, there is a growing demand for Small Language Models (SLMs) that are efficient for edge deployment and low-latency applications. Traditionally, SLMs are either trained from scratch or derived via structured pruning. However, training from scratch often hits a performance ceiling defined by scaling laws, while pruning—which involves discarding "unimportant" neurons or layers—can lead to significant information loss.
LEMON (Linear Parameter Fusion) addresses these challenges by proposing a "fusion" approach instead of the traditional "search and destroy" pruning method. Rather than treating parameters as binary (keep or discard), LEMON learns to compress a large model into a smaller one by fusing parameters, thereby preserving functional knowledge more effectively from the entire source model.
Methodology: From Pruning to Fusion
The core innovation of LEMON is replacing binary pruning with learnable linear operators that transform large parameter matrices into smaller ones. This ensures that the resulting SLM inherits a distilled version of the larger model's entire weight distribution.
Linear Parameter Fusion
LEMON decomposes the fusion process into two primary operations:
- Vertical Fusion (Layer Operator): LLMs often exhibit modularity where adjacent layers perform similar functions. LEMON learns a mapping matrix ($\gamma$) to combine multiple layers from the source model into a single target layer.
- Horizontal Fusion (Dimension Operator): To reduce hidden dimensions (e.g., shrinking a 4096-dimension vector to 2560) without a parameter explosion, LEMON utilizes Kronecker products to create efficient "Dimension Operators."
- Controllable Receptive Fields: A "granularity" parameter ensures structural integrity, such as maintaining the distinctness of attention heads during the fusion process.
The framework operates in two stages: an initial phase where fusion operators are learned using a minimal dataset (0.26B tokens), followed by standard continued pre-training of the resulting smaller model.
Empirical Motivation: Functional Clusters
A critical observation justifying the fusion approach is the discovery of functional clusters within LLMs. Using Centered Kernel Alignment (CKA), the authors found that layers, particularly in the middle and deeper sections of the model, exhibit high similarity.
This high similarity suggests that layers are not unique entities but part of functional groups, making them ideal candidates for fusion rather than arbitrary deletion. By fusing these clusters, the small model inherits the collective knowledge of the group.
Evaluation and Results
LEMON was evaluated using LLaMA 2-7B as a source to produce 1.3B and 2.7B models. Testing across 11 benchmarks, including MMLU, HellaSwag, and ARC, revealed several key findings:
- Superior Performance: LEMON-2.7B achieved an average score of 57.6 across benchmarks, outperforming the pruning-based Sheared-LLaMA-2.7B (56.7) and the scratch-trained INCITE-Base-3B (54.7).
- High Efficiency: The initialization budget for LEMON is remarkably low. LEMON-2.7B reached the perplexity of state-of-the-art pruning baselines in 62.5% fewer steps, representing a significant saving in training resources.
- Resource Savings: Compared to models trained from scratch like Pythia, LEMON required only 5.26% of the total pre-training tokens to achieve superior results.
Conclusion and Limitations
LEMON demonstrates that "reviving" a smaller model through parameter fusion is significantly more effective than traditional pruning. By treating all parameters as valuable and learning how to combine them, LEMON provides a high-quality starting point for SLM training, drastically reducing the required computational budget.
While effective, the current study is limited to linear fusion; non-linear methods might capture more complex relationships but would increase optimization difficulty. Additionally, while the final model is small, the fusion learning phase still requires significant memory to keep the large source model in the loop. Future work will explore scaling these patterns to models beyond the 7B parameter range.
Summary updated