Abstract
Traditional scaling laws for centralized machine learning suggest that increasing model parameters and data volume consistently improves performance. However, these laws do not account for the unique constraints of Federated Learning (FL), such as data decentralization and heterogeneity. This paper investigates scaling laws within the FL paradigm, deriving both theoretical and empirical evidence that the optimal model size decreases as the number of clients increases for a fixed total compute budget. We demonstrate that this relationship follows a negative power law, providing a principled framework for model selection in privacy-preserving, distributed environments.
Introduction
The prevailing "bigger is better" philosophy in AI, exemplified by models like GPT-3, is predicated on centralized training where data is unified. Federated Learning (FL) shifts this landscape by training models across decentralized edge devices. This decentralization introduces a "generalization tax," where the distributed nature of data impacts the efficiency of the learning process. Understanding how to "right-size" models for FL is critical for optimizing performance under resource constraints.
Problem Statement
In centralized training, scaling laws allow practitioners to predict the optimal model size for a given compute budget. In FL, the number of clients ($n$) and the heterogeneity of their local data ($\gamma$) become critical variables. Until now, there has been no formal method to determine if a massive model is appropriate when trained across thousands of devices or if a smaller architecture would yield better generalization. This research addresses this gap by quantifying the impact of data decentralization on the ideal model size.
Methodology
The study employs a dual approach:
- Theoretical Analysis: Utilizing the PAC-Bayes framework, the authors derive an upper bound for generalization error in Federated SGD. They identify a closed-form approximation for the optimal model size ($d^*$) that minimizes this error.
- Empirical Validation: The researchers trained Vision Transformers (ViT) and ResNets of varying sizes (10 distinct scales) across federated scenarios with client counts ranging from 1 to 150.
Datasets and Evaluation
Models were pre-trained on Mini-ImageNet using a Masked Auto-encoder (MAE) approach. Evaluation was conducted on CIFAR-100 and ImageNet using linear probing (LP) and full fine-tuning (FT) to assess the quality of the learned representations.
Key Insights: The Negative Power Law
The central finding is that for a fixed total training compute, the optimal model size follows a negative power law relative to the number of clients. As the "distributed-ness" of the system increases, the capacity of the model that can be effectively trained by each client decreases.
This phenomenon suggests that the average compute per client is the primary determinant of the optimal model size. If individual devices have limited data or compute, a large model is prone to overfitting or poor generalization, regardless of the total aggregate data in the network.
Results and Discussion
The empirical data strongly supports the theoretical derivations:
- Mathematical Consistency: The relationship $d^*_{Fed} \approx \frac{\rho}{n^{\gamma-1}} d^*_{Cen}$ was validated, where $\gamma > 1$ represents data heterogeneity.
- Empirical Fit: Log-log plots of optimal model size versus client count showed an $R^2 > 0.88$ across benchmarks.
- Architecture Agnosticism: The scaling trends were consistent across both Transformer-based (ViT) and Convolutional (ResNet) architectures, as well as different training regimes like fine-tuning.
Limitations and Future Work
While providing a foundational framework, this work has specific boundaries:
- Linear Probing Focus: Much of the empirical validation relies on linear probing, which may not always perfectly capture full fine-tuning dynamics.
- Fixed Compute Constraint: The current model assumes a static total compute budget. Future research should explore "expanding budget" scenarios where adding clients also increases the total available data and compute.
- Local Convexity: Theoretical proofs assume the model operates in a locally convex regime, which may not fully describe the initial phases of training very large models from scratch.
Conclusion
This research provides the first principled "manual" for sizing models in Federated Learning. By recognizing the negative power law relationship between client count and optimal model size, practitioners can avoid the inefficiencies of over-parameterization in decentralized systems and achieve better generalization on edge devices.
Summary updated