Abstract
As machine learning models increase in scale, training them on resource-constrained edge devices becomes increasingly difficult. Sub-model training has emerged as a practical solution, allowing clients to optimize only a fraction of the global model. This paper provides the first rigorous theoretical analysis of the convergence and stability of distributed sub-model training, introducing a "shuffled rolling" strategy that improves performance over standard random sampling and demonstrating how sub-model updates act as an implicit regularizer to enhance generalization.
Introduction
The rapid growth of model parameters presents a significant barrier to on-device learning. In standard Federated Learning (FL) frameworks like FedAvg, every participating client must download, optimize, and upload the entire global model. For smartphones or IoT sensors, the memory and computational overhead of billion-parameter architectures is often prohibitive.
To address this, researchers have proposed "partial training" or sub-model training, where clients update only a subset of the model's weights. While empirical frameworks like HeteroFL and FedRolex have shown promise, the mathematical foundations of these methods—specifically regarding their convergence guarantees and their impact on model stability—have remained largely unexplored until now.
Methodology: Masking Strategies
The researchers analyzed two primary strategies for selecting sub-models during the training process:
- Randomly Masked FedAvg: In this approach, the server generates a random Bernoulli mask for each client. Each parameter has a fixed probability of being selected for an update in any given round, effectively "flipping a coin" for every weight in the network.
- Rolling Masked FedAvg: This proposed strategy partitions the global model into $R$ distinct pieces. At the start of each epoch, the server shuffles these pieces and assigns them to clients sequentially. This ensures that every parameter is updated systematically within a fixed cycle, preventing the "stale parameter" problem associated with random sampling.
Technically, the study establishes that sub-model training does not optimize the original objective function $F(w)$ directly. Instead, it optimizes an "alternative objective" induced by the masking process, which the authors characterize through convergence rates in both convex and non-convex settings.
Theoretical Insights: Stability and Generalization
A core contribution of this work is the link between sub-model training and algorithmic stability. The authors demonstrate that updating only a subset of weights acts as a form of implicit regularization. By limiting the influence of any single data point on the total model update, the training process becomes more stable.
This stability is mathematically shown to reduce the generalization gap—the difference between training performance and test performance. In high-heterogeneity environments typical of federated learning, this regularization helps prevent the global model from overfitting to the local distributions of specific clients.
Experimental Evaluation
The researchers evaluated the proposed methods using CIFAR-10 and CIFAR-100 datasets on a modified ResNet18 architecture. To better suit the federated environment, standard Batch Normalization was replaced with Static Batch Normalization. Data heterogeneity was simulated using Dirichlet distributions to represent real-world non-IID (Independent and Identically Distributed) scenarios.
Key Results
- Convergence Rates: Under strongly convex settings, both masking algorithms achieve an $O(1/T)$ convergence rate, though they retain a small, controllable residual error compared to full-model training.
- Superiority of Rolling Masking: On the CIFAR-100 benchmark with high data heterogeneity, the Rolling Masking approach achieved 35.30% accuracy, outperforming the static masking used in HeteroFL, which reached 34.37%.
- Generalization Benefits: The "Global Accuracy Diff" (the gap between training and test accuracy) was significantly lower for sub-model training. Under high heterogeneity, random masking showed a gap of 0.384, whereas full-model training exhibited a much larger gap of 1.488, confirming the stabilizing effect of the masking process.
Discussion and Future Work
While this paper provides a robust foundation for sub-model training, several challenges remain. The current theoretical analysis relies on assumptions of strong convexity and bounded gradients, which may not fully capture the complexities of deep neural networks like Transformers. Additionally, while sub-models offer a degree of privacy by ensuring clients never see the full model, they may introduce new security vulnerabilities, such as allowing high-capacity clients to exert disproportionate influence on the global parameters.
Future research directions include the development of masking schemes that are both privacy-preserving and robust against adversarial attacks, as well as extending the stability analysis to more complex, non-convex architectures.
Summary updated