Introduction
Large Language Models (LLMs) have revolutionized natural language processing but remain computationally prohibitive for edge devices like smartphones or IoT hardware. While Small Language Models (SLMs) offer a lighter alternative, they often lack the robustness and generalization of their larger counterparts. Furthermore, standard Federated Learning (FL) assumes a "one-size-fits-all" architecture, which is impractical given the hardware heterogeneity of real-world devices.
To bridge this gap, we present LEGO (Language ModEl BuildinG BlOcks), a framework that prunes LLMs into diverse, smaller building blocks. This allows devices with varying computational budgets to collaborate on a single, high-quality global model without sacrificing privacy or performance.
The LEGO Framework
The LEGO approach consists of two primary stages designed to handle hardware and data heterogeneity:
- Model Pruning: Using SparseGPT, we extract SLMs with varying sparsity levels (e.g., 25%, 50%, 75%) from a base LLaMA-7B model. This creates a library of "building blocks" tailored to different hardware constraints.
- Heterogeneous Aggregation (HeteAgg): In a typical FL setup, weights from identical models are averaged. LEGO utilizes LoRA (Low-Rank Adaptation) adapters. When clients return sparse updates, LEGO applies a mask based on the sparsity pattern. The HeteAgg algorithm then aggregates these updates into a dense global model by averaging only the non-zero parameters where they overlap.
This mechanism ensures that even a highly sparse model on a resource-constrained device can contribute its learned knowledge to the full-sized global model.
Experimental Setup
- Base Model: LLaMA-7B.
- Fine-tuning Data: The databricks-dolly-15k dataset, covering eight capability domains such as creative writing and information extraction.
- Evaluation: Models were benchmarked using the EleutherAI Language Model Evaluation Harness on HellaSwag, MMLU, SciQ, and ARC.
Key Insights
SLMs as Efficient Learners
Pruning appears to act similarly to dropout, forcing the model to develop stronger, more general representations within a limited parameter budget. Consequently, SLMs can sometimes be more efficient learners for specific tasks than their dense counterparts.
Mitigating Data Heterogeneity
LEGO excels in non-i.i.d. (non-independent and identically distributed) scenarios where data is task-dependent. By allowing SLMs to specialize in specific domains before aggregation, the final recombined LLM becomes more robust than one trained on a uniform global shuffle.
Efficiency Gains
The use of pruned models resulted in up to a 1.6x speedup in inference and a 1.4x speedup in fine-tuning compared to full-sized models, significantly lowering the barrier for on-device AI.
Results and Discussion
Empirical results demonstrate that the LEGO-recombined LLM achieves an accuracy of 0.571 on task-dependent data, outperforming the standard Federated Instruction Tuning (FedIT) baseline of 0.563.
As shown in the performance analysis, the global model remains stable and high-performing even when combining "differently shaped" blocks. The experiments confirm that smaller models effectively transfer knowledge to larger models during the aggregation phase, validating the modular "building block" philosophy.
Conclusion and Future Work
While LEGO shows great promise, some challenges remain. The separate aggregation of LoRA modules $A$ and $B$ introduces a degree of mathematical noise ($\sum A \times \sum B \neq \sum (A \times B)$). Additionally, LEGO's performance is currently tied to the efficacy of pruning techniques like SparseGPT. Future research will explore scaling this approach to larger models (70B+) and further refining the aggregation algorithms to minimize noise.
Summary updated