Scaling Law Analysis in Federated Learning: How to Select the Optimal Model Size?

Xuanyu Chen, Nan Yang, Shuai Wang, Dong YuanarXiv 2025 · DOI · arXiv

Abstract

Traditional scaling laws for centralized machine learning suggest that increasing model parameters and data volume consistently improves performance. However, these laws do not account for the unique constraints of Federated Learning (FL), such as data decentralization and heterogeneity. This paper investigates scaling laws within the FL paradigm, deriving both theoretical and empirical evidence that the optimal model size decreases as the number of clients increases for a fixed total compute budget. We demonstrate that this relationship follows a negative power law, providing a principled framework for model selection in privacy-preserving, distributed environments.

Introduction

The prevailing "bigger is better" philosophy in AI, exemplified by models like GPT-3, is predicated on centralized training where data is unified. Federated Learning (FL) shifts this landscape by training models across decentralized edge devices. This decentralization introduces a "generalization tax," where the distributed nature of data impacts the efficiency of the learning process. Understanding how to "right-size" models for FL is critical for optimizing performance under resource constraints.

Problem Statement

In centralized training, scaling laws allow practitioners to predict the optimal model size for a given compute budget. In FL, the number of clients ($n$) and the heterogeneity of their local data ($\gamma$) become critical variables. Until now, there has been no formal method to determine if a massive model is appropriate when trained across thousands of devices or if a smaller architecture would yield better generalization. This research addresses this gap by quantifying the impact of data decentralization on the ideal model size.

Methodology

The study employs a dual approach:

  1. Theoretical Analysis: Utilizing the PAC-Bayes framework, the authors derive an upper bound for generalization error in Federated SGD. They identify a closed-form approximation for the optimal model size ($d^*$) that minimizes this error.
  2. Empirical Validation: The researchers trained Vision Transformers (ViT) and ResNets of varying sizes (10 distinct scales) across federated scenarios with client counts ranging from 1 to 150.

Datasets and Evaluation

Models were pre-trained on Mini-ImageNet using a Masked Auto-encoder (MAE) approach. Evaluation was conducted on CIFAR-100 and ImageNet using linear probing (LP) and full fine-tuning (FT) to assess the quality of the learned representations.

Key Insights: The Negative Power Law

The central finding is that for a fixed total training compute, the optimal model size follows a negative power law relative to the number of clients. As the "distributed-ness" of the system increases, the capacity of the model that can be effectively trained by each client decreases.

Figure 1: Scaling laws for ViT on CIFAR-100 and ImageNet. Left panels show accuracy vs. model size for different client counts; right panels illustrate the negative power law relationship between optimal model size and the number of clients.

This phenomenon suggests that the average compute per client is the primary determinant of the optimal model size. If individual devices have limited data or compute, a large model is prone to overfitting or poor generalization, regardless of the total aggregate data in the network.

Results and Discussion

The empirical data strongly supports the theoretical derivations:

  • Mathematical Consistency: The relationship $d^*_{Fed} \approx \frac{\rho}{n^{\gamma-1}} d^*_{Cen}$ was validated, where $\gamma > 1$ represents data heterogeneity.
  • Empirical Fit: Log-log plots of optimal model size versus client count showed an $R^2 > 0.88$ across benchmarks.
  • Architecture Agnosticism: The scaling trends were consistent across both Transformer-based (ViT) and Convolutional (ResNet) architectures, as well as different training regimes like fine-tuning.
Figure 5: Validation of scaling laws for ResNet architectures using full fine-tuning on CIFAR-100. The negative power law trend persists across different architectures and training objectives.

Limitations and Future Work

While providing a foundational framework, this work has specific boundaries:

  • Linear Probing Focus: Much of the empirical validation relies on linear probing, which may not always perfectly capture full fine-tuning dynamics.
  • Fixed Compute Constraint: The current model assumes a static total compute budget. Future research should explore "expanding budget" scenarios where adding clients also increases the total available data and compute.
  • Local Convexity: Theoretical proofs assume the model operates in a locally convex regime, which may not fully describe the initial phases of training very large models from scratch.

Conclusion

This research provides the first principled "manual" for sizing models in Federated Learning. By recognizing the negative power law relationship between client count and optimal model size, practitioners can avoid the inefficiencies of over-parameterization in decentralized systems and achieve better generalization on edge devices.

Summary updated

Abstract

The recent success of large language models (LLMs) has sparked a growing interest in training large-scale models. As the model size continues to scale, concerns are growing about the depletion of high-quality, well-curated training data. This has led practitioners to explore training approaches like Federated Learning (FL), which can leverage the abundant data on edge devices while maintaining privacy. However, the decentralization of training datasets in FL introduces challenges to scaling large models, a topic that remains under-explored. This paper fills this gap and provides qualitative insights on generalizing the previous model scaling experience to federated learning scenarios. Specifically, we derive a PAC-Bayes (Probably Approximately Correct Bayesian) upper bound for the generalization error of models trained with stochastic algorithms in federated settings and quantify the impact of distributed training data on the optimal model size by finding the analytic solution of model size that minimizes this bound. Our theoretical results demonstrate that the optimal model size has a negative power law relationship with the number of clients if the total training compute is unchanged. Besides, we also find that switching to FL with the same training compute will inevitably reduce the upper bound of generalization performance that the model can achieve through training, and that estimating the optimal model size in federated scenarios should depend on the average training compute across clients. Furthermore, we also empirically validate the correctness of our results with extensive training runs on different models, network settings, and datasets.
BibTeX
@misc{chen2025scaling,
  author = {Xuanyu Chen and Nan Yang and Shuai Wang and Dong Yuan},
  title = {{Scaling Law Analysis in Federated Learning: How to Select the Optimal Model Size?}},
  year = {2025},
  eprint = {2511.12188},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.12188},
}
Figure