Federated Customization of Large Models: Approaches, Experiments, and Insights

Yuchuan Ye, Ming Ding, Youjia Chen, Peng Cheng, Dusit Niyato

Abstract

In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.

Introduction: The Challenge of Customizing Large Models

Large Models (LMs) such as GPT-4 and LLaMA have demonstrated remarkable capabilities across various domains. However, to be truly effective for niche, domain-specific tasks—such as processing medical records or internal corporate documents—these models often require customization on private data. Traditionally, this is achieved through fine-tuning, but when data is siloed across multiple clients due to privacy concerns, standard centralized training becomes impossible.

Federated Learning (FL) offers a solution by allowing collaborative training without sharing raw data. Yet, applying FL to LMs introduces significant hurdles: edge devices often lack the compute and memory to train billions of parameters, and the inherent "Non-IID" (non-identically and independently distributed) nature of decentralized data can cause training to diverge. This paper shifts the focus from training foundation models from scratch to the more practical task of Federated Customization.

Federated Customization Strategies

The researchers evaluated six primary strategies for adapting large models within a federated framework. These range from resource-intensive full fine-tuning to lightweight prompt-based methods.

Overview of various federated customization architectures, including fine-tuning, RAG, and prefix-tuning.
  1. Federated Full Fine-Tuning: Retraining the entire model, which is computationally prohibitive for most FL clients.
  2. Federated Efficient Fine-Tuning: Updating only a small portion of the model (e.g., using Adapters or LoRA).
  3. Federated Prompt Engineering: Learning optimal prompts to guide the frozen LM.
  4. Federated RAG: Using federated retrieval to augment the LM with private knowledge bases.
  5. Federated Knowledge Distillation: Training smaller "student" models using the large LM as a "teacher."
  6. Federated Prefix-Tuning (FPT): Prepending learnable vectors to the input while keeping the backbone frozen.

Comparative Analysis of Costs and Features

Each method presents a different trade-off between performance, communication overhead, and local compute requirements. As shown in the table below, parameter-efficient methods like prefix-tuning significantly lower the "entry fee" for resource-constrained clients by keeping the massive LM backbone frozen.

Comparison of federated customization methods based on communication and computing costs.

The Innovation: Federated Prefix-Tuning (FPT)

The core contribution of this work is the introduction of Federated Prefix-Tuning (FPT). In standard prefix-tuning, a set of learnable vectors is prepended to the model's input. However, raw prefix vectors are unstructured and difficult to aggregate effectively across diverse clients.

FPT introduces a Prefix Optimizer Network (a 3-layer MLP) on each client. Instead of sharing raw vectors, clients train and share the weights of this MLP. The server aggregates these weights using FedAvg. This structured parameter space makes the collaborative learning process more stable and allows the model to capture underlying task patterns rather than just memorizing token sequences.

Experimental Evaluation

The authors tested FPT on two major "table-to-text" generation benchmarks: E2E (~42,000 restaurant descriptions) and DART (~82,000 complex examples from WikiSQL). Experiments covered both IID and Non-IID scenarios to simulate real-world data fragmentation.

Key Results

  • Near-Centralized Performance: FPT achieved results remarkably close to centralized training. On the E2E dataset, FPT reached a BLEU score of 68.91, nearly matching the centralized benchmark of 69.61.
  • Efficiency and Scalability: By keeping the LM backbone frozen, FPT reduced trainable parameters by over 90% compared to full fine-tuning (e.g., 25M vs. 345M parameters). Performance gains were consistent as the backbone model size increased from GPT-2 Medium to GPT-2 Large.
  • Robustness to Data Heterogeneity: FPT proved resilient to Non-IID data. The shared frozen backbone provides a stable foundation that prevents the global model from collapsing, even when client data distributions vary significantly.

Conclusion and Future Work

Federated Prefix-Tuning demonstrates that high-performance customization of large models is possible even under strict privacy and resource constraints. By focusing on optimizing a small prefix-generating network rather than the entire model, FPT achieves a balance of efficiency and accuracy.

Future research will address the "loading" problem—the requirement for clients to still hold the massive frozen backbone in memory—and explore the integration of formal privacy guarantees like Differential Privacy. Additionally, extending these techniques to multi-modal and multi-task environments remains a promising frontier.

Summary updated

Abstract

In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.
BibTeX
@article{ye2026federated,
  author = {Yuchuan Ye and Ming Ding and Youjia Chen and Peng Cheng and Dusit Niyato},
  title = {{Federated Customization of Large Models: Approaches, Experiments, and Insights}},
  journal = {IEEE Network},
  year = {2026},
}
Figure