LEGO: Language Model Building Blocks

Shrenik Bhansali, Alwin Jin, Tyler Lizzo, Larry HeckarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) have revolutionized natural language processing but remain computationally prohibitive for edge devices like smartphones or IoT hardware. While Small Language Models (SLMs) offer a lighter alternative, they often lack the robustness and generalization of their larger counterparts. Furthermore, standard Federated Learning (FL) assumes a "one-size-fits-all" architecture, which is impractical given the hardware heterogeneity of real-world devices.

To bridge this gap, we present LEGO (Language ModEl BuildinG BlOcks), a framework that prunes LLMs into diverse, smaller building blocks. This allows devices with varying computational budgets to collaborate on a single, high-quality global model without sacrificing privacy or performance.

The LEGO Framework

The LEGO approach consists of two primary stages designed to handle hardware and data heterogeneity:

  1. Model Pruning: Using SparseGPT, we extract SLMs with varying sparsity levels (e.g., 25%, 50%, 75%) from a base LLaMA-7B model. This creates a library of "building blocks" tailored to different hardware constraints.
  2. Heterogeneous Aggregation (HeteAgg): In a typical FL setup, weights from identical models are averaged. LEGO utilizes LoRA (Low-Rank Adaptation) adapters. When clients return sparse updates, LEGO applies a mask based on the sparsity pattern. The HeteAgg algorithm then aggregates these updates into a dense global model by averaging only the non-zero parameters where they overlap.
The LEGO workflow: Pruning a base LLM into diverse SLMs, assigning them to heterogeneous clients, and aggregating local updates into a unified global model.

This mechanism ensures that even a highly sparse model on a resource-constrained device can contribute its learned knowledge to the full-sized global model.

Experimental Setup

  • Base Model: LLaMA-7B.
  • Fine-tuning Data: The databricks-dolly-15k dataset, covering eight capability domains such as creative writing and information extraction.
  • Evaluation: Models were benchmarked using the EleutherAI Language Model Evaluation Harness on HellaSwag, MMLU, SciQ, and ARC.

Key Insights

SLMs as Efficient Learners

Pruning appears to act similarly to dropout, forcing the model to develop stronger, more general representations within a limited parameter budget. Consequently, SLMs can sometimes be more efficient learners for specific tasks than their dense counterparts.

Mitigating Data Heterogeneity

LEGO excels in non-i.i.d. (non-independent and identically distributed) scenarios where data is task-dependent. By allowing SLMs to specialize in specific domains before aggregation, the final recombined LLM becomes more robust than one trained on a uniform global shuffle.

Efficiency Gains

The use of pruned models resulted in up to a 1.6x speedup in inference and a 1.4x speedup in fine-tuning compared to full-sized models, significantly lowering the barrier for on-device AI.

Results and Discussion

Empirical results demonstrate that the LEGO-recombined LLM achieves an accuracy of 0.571 on task-dependent data, outperforming the standard Federated Instruction Tuning (FedIT) baseline of 0.563.

Performance of LEGO across different sparsity levels (0%, 25%, 50%, 75%) compared to full aggregation and fine-tuned baselines.

As shown in the performance analysis, the global model remains stable and high-performing even when combining "differently shaped" blocks. The experiments confirm that smaller models effectively transfer knowledge to larger models during the aggregation phase, validating the modular "building block" philosophy.

Conclusion and Future Work

While LEGO shows great promise, some challenges remain. The separate aggregation of LoRA modules $A$ and $B$ introduces a degree of mathematical noise ($\sum A \times \sum B \neq \sum (A \times B)$). Additionally, LEGO's performance is currently tied to the efficacy of pruning techniques like SparseGPT. Future research will explore scaling this approach to larger models (70B+) and further refining the aggregation algorithms to minimize noise.

Summary updated

Abstract

Large language models (LLMs) are essential in natural language processing (NLP) but are costly in data collection, pre-training, fine-tuning, and inference. Task-specific small language models (SLMs) offer a cheaper alternative but lack robustness and generalization. This paper proposes LEGO, a novel technique to extract SLMs from an LLM and recombine them. Using state-of-the-art LLM pruning strategies, we can create task- and user-specific SLM building blocks that are efficient for fine-tuning and inference while also preserving user data privacy. LEGO utilizes Federated Learning and a novel aggregation scheme for the LLM reconstruction, maintaining robustness without high costs and preserving user data privacy. We experimentally demonstrate the versatility of LEGO, showing its ability to enable model heterogeneity and mitigate the effects of data heterogeneity while maintaining LLM robustness.
BibTeX
@misc{bhansali2024lego,
  author = {Shrenik Bhansali and Alwin Jin and Tyler Lizzo and Larry Heck},
  title = {{LEGO: Language Model Building Blocks}},
  year = {2024},
  eprint = {2410.18287},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.18287},
}
Figure