Federated Learning of Large Models at the Edge via Principal Sub-Model Training

Yue Niu, Saurav Prakash, Souvik Kundu, Sunwoo Lee, Salman Avestimehr

TL;DR

The paper introduces PriSM (Principal Sub-Model training), a novel federated learning framework designed for ultra-resource-constrained edge devices that cannot afford to train a full large AI model. By probabilistically sampling principal kernels in an orthogonal weight space, PriSM distributes tiny, memory-efficient sub-models to edge clients while collaboratively assembling a full-capacity global model on the server without compromising data privacy.

What Problem Does This Paper Address?

Federated Learning (FL) allows decentralized devices to collaboratively train a shared model while keeping data local and private. However, modern deep learning models (like Transformers and deep CNNs) are massive, creating severe computation, memory, and communication bottlenecks for resource-constrained edge devices (e.g., IoT sensors, smart home appliances).

Prior attempts to solve this either:

  1. Assume a heterogeneous client mix: Some powerful clients train the full model while weaker ones train smaller sub-models. This fails if all edge devices are weak.
  2. Offload training via Split Learning: Clients process early layers and send intermediate representations to a central server. This breaks the core promise of FL by leaking private data through intermediate activations or labels.

PriSM addresses a highly realistic and challenging cross-device FL setting: where no single client is capable of training the full model, and no client is willing to share intermediate data with the server.

Methods

PriSM achieves full-model performance through a three-step pipeline executed each communication round:

  1. Orthogonal Space Decomposition: The server converts standard model layers (like convolutions or linear layers) into an orthogonal space using Singular Value Decomposition (SVD). This breaks the weights down into principal kernels, where each kernel's importance is directly quantified by its corresponding singular value ($\sigma$).
  2. Importance-Aware Probabilistic Sampling: Instead of deterministically keeping only the top-$k$ kernels (which permanently discards model capacity), PriSM samples a subset of $r$ kernels for each client. The sampling probability is proportional to the kernel's singular value ($p_i \propto \sigma_i^\kappa$). This ensures that vital features are trained frequently, while less critical features are still explored and updated across different clients over time.
  3. Memory-Efficient Feature Learning: To alleviate activation memory bottlenecks on edge devices, PriSM splits the sampled convolution layers into two sublayers ($Conv_U$ and $Conv_V$) and forces $Conv_U$ to compute only a subset of output channels, significantly dropping the local memory footprint.

After local training, the server aggregates the sub-models, reconstructs the full model, and refreshes the orthogonal space via SVD for the next round.

The PriSM sub-model creation pipeline, mapping original kernels to an orthogonal space via SVD before applying importance-aware sampling.

Datasets

The authors evaluated PriSM across various architectures and modalities:

  • Vision: ResNet-18 and DeiT (Data-efficient Image Transformer) trained on CIFAR-10.
  • Edge Vision: A standard CNN trained on FEMNIST (Federated Extended MNIST).
  • Language: An LSTM model trained on the IMDB movie review dataset for sentiment analysis.

To simulate realistic edge environments, datasets were split into both Independent and Identically Distributed (i.i.d.) and non-Independent and Identically Distributed (non-i.i.d.) settings using a Dirichlet distribution ($\alpha = 1.0$ and $\alpha = 0.1$).

Key Insights and Ideas

  • The 50% Capacity Floor: The authors discovered that even late in FL training, neural network layers require at least half of their principal kernels to maintain accurate approximations. Deterministic truncation methods (like FedHM/OrthDrop) permanently drop lower-ranked kernels, severely damaging final model capacity. PriSM's probabilistic sampling ensures full coverage over multiple communication rounds.
  • Orthogonal Space Simplifies Importance: Ranking feature importance in standard weight space is notoriously difficult. Decomposing weights into an orthogonal space allows singular values to serve as an elegant, clean proxy for feature importance (e.g., separating structural outlines from fine textures).
  • Activation Memory is the Real Bottleneck: Shrinking model weights reduces communication costs, but edge devices often crash due to peak activation memory during the forward/backward pass. PriSM's channel-reduction technique directly targets and mitigates this activation memory overhead.

Key Results

  • Substantial Accuracy Gains: Under highly constrained non-i.i.d. settings ($\alpha = 0.1$), PriSM outperformed competitive baselines (OrthDrop, OrigDrop, SplitMix) by up to 10% to 14% in final validation accuracy.
  • Minimal Degradation with Tiny Sub-models: On ResNet-18/CIFAR-10, when clients trained tiny 0.2× sub-models (representing a mere ~5% of full server model parameters and computations), PriSM incurred only a ~2% accuracy drop compared to ideal, unconstrained full-model training.
  • Negligible Server Overhead: Profiling the runtime breakdown revealed that the server-side SVD computation is highly efficient, taking less than 1 second per round, compared to the ~36 seconds spent on local client training.

Limitations and Future Work

  • Scalability to LLMs: While the server-side SVD overhead is negligible for ResNet-18 and smaller Transformers, performing SVD every communication round on massive Large Language Models (LLMs) with tens of billions of parameters could introduce significant computational bottlenecks at the server level.
  • Hyperparameter Tuning: PriSM introduces a smoothing factor ($\kappa$) to control the sampling distribution. The optimal value of $\kappa$ varies depending on the target sub-model size, requiring careful tuning.
  • Future Directions: The authors suggest exploring adaptive rank allocation, where sub-model sizes are dynamically tailored to each client's real-time hardware capacity rather than being fixed uniformly.

Summary updated

Abstract

Federated Learning (FL) is emerging as a popular, promising decentralized learning framework that enables collaborative training among clients, with no need to share private data between them or to a centralized server. However, considering many edge clients do not have sufficient computing, memory, or communication capabilities, federated learning of large models still faces significant bottlenecks. To keep such weak but crucial clients in the loop, prior works either consider a heterogeneous-client setting where clients train models with different sizes; or offload training to the server. However, the heterogeneous-client setting requires some clients to train full model, which is not aligned with the resource-constrained setting; while the latter ones break privacy promises in FL when sharing intermediate representations or labels with the server. To overcome these limitations, in this work, we formulate a realistic, but much less explored, cross-device FL setting in which no client can train a full large model nor is willing to share any intermediate information with the remote server. Under such a formulation, we develop a principal sub-model (PriSM) training methodology to collaboratively train a full large model, while assigning each client a small sub-model that is a probabilistic low-rank approximation to the full server model. When creating sub-models, PriSM first performs a principal kernel analysis in the orthogonal kernel space to obtain importance of each kernel. Then, PriSM adopts a novel importance-aware sampling process to select a subset of kernels (i.e., a kernel with high importance is assigned with a higher sampling probability). This sampling process ensures each sub-model is still a low-rank approximation to the full model, while all sub-models together achieve nearly full coverage on the principal kernels.
BibTeX
@inproceedings{niu2023federated,
  author = {Yue Niu and Saurav Prakash and Souvik Kundu and Sunwoo Lee and Salman Avestimehr},
  title = {{Federated Learning of Large Models at the Edge via Principal Sub-Model Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=8Vxuz_PJNus},
}
Figure