Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model

Gautam JajooAAAI 2025 · DOI

Abstract

This paper, presented at AAAI-25, introduces a novel Federated Learning (FL) framework designed to bridge the gap between resource-constrained edge devices and massive central servers. By employing a "Specialist-Generalist" architecture, the system allows small, task-specific "student" models to transfer knowledge to a "large hungry" server model via task vectors and linear projections. This approach addresses the challenges of model heterogeneity and privacy-preserving knowledge distillation without requiring shared public datasets, ensuring both computational efficiency and data privacy.

Introduction

Standard Federated Learning typically assumes homogeneity, where every participant is expected to run the same model architecture. However, in real-world scenarios, hardware capabilities vary significantly; a smartphone cannot run the same 70-billion parameter model as a cloud server. While Heterogeneous Federated Learning (HFL) exists, integrating models of vastly different sizes remains a significant challenge. Furthermore, existing knowledge distillation techniques often rely on shared public datasets, which can compromise the privacy goals of a decentralized setup. This research proposes a method for "tiny" models on edge devices to effectively "teach" a "giant" model on a central server.

Methodology

The proposed "Specialist-Generalist" architecture consists of two primary components:

  • Client Side (The Specialists): Clients utilize lightweight models (e.g., Bloom-560m) fine-tuned on local, private data for specific tasks. Instead of transmitting the entire model, clients generate Task Vectors—the mathematical difference between pre-trained base weights and fine-tuned weights—capturing the essential direction and magnitude of local learning.
  • Server Side (The Hungry Model): The server hosts a much larger model (e.g., Bloom-7b1). To bridge the dimensionality gap between the small client updates and the large server weight space, the framework employs learned linear transformation matrices. These projections map low-dimensional updates into the higher-dimensional space of the server model.

Aggregation is handled through a weighted formula that accounts for update quality and data heterogeneity. Additionally, the system uses Layer-wise Adaptive Learning Rates (LARS) to adjust learning intensity based on weight magnitude across different architectures.

Experimental Setup

The framework is evaluated across a comprehensive benchmark of NLP domains, ranging from generative tasks like summarization to classification and reasoning. In addition to standard performance metrics, the study tracks system-level indicators to ensure viability for edge computing environments.

Table 1: Overview of task types, datasets, and evaluation metrics used in the study.

Key Insights and Results

  • The "Hungry Model" Paradigm: The server acts as a generalist that consumes specialized insights from diverse clients, rather than just a manager averaging weights.
  • Task Arithmetic in FL: Utilizing task vectors allows for more nuanced updates than simple gradient averaging, particularly when architectures are mismatched.
  • Efficiency and Scalability: By communicating only task vectors and leveraging small client models, the system significantly reduces bandwidth and computational barriers. The author expects the large server model to demonstrate superior generalization by aggregating these diverse, task-specific insights.

Limitations and Future Work

While promising, the framework faces challenges in aggregation complexity, as simple averaging is insufficient for reconciling disparate parameter counts. There is also a risk of catastrophic forgetting, where the server model might lose general knowledge while learning specific tasks; the author suggests continual learning techniques as a potential solution. Finally, the sufficiency of linear projections for mapping complex, non-linear weight updates remains an open research question to be explored in future implementations.

Summary updated

BibTeX
@inproceedings{jajoo2025federated,
  author = {Gautam Jajoo},
  title = {{Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35332},
}
Figure