Rolling Sub-model Extraction for Heterogeneous Federated Learning

This direction is merged into another direction: architectural homogeneity constraint (system heterogeneity). Its last version stays readable and citable here.

Version 1 of 110 papers18 conceptsfrozen 2026-10-04

In standard Federated Learning (FL), we assume a "one size fits all" architecture: every device, from a high-end server to a budget smartphone, trains the exact same global model. In practice, this creates a hardware wall. Large-scale Transformers or deep ResNets often exceed the memory and computational limits of edge devices, leading to the "straggler" problem where the entire training process waits for the slowest, most resource-constrained participants.

To bypass this, researchers have moved toward Model-Heterogeneous Federated Learning, where clients train smaller "sub-models" extracted from the larger global model. The challenge is ensuring that these fragmented updates actually lead to a coherent, high-performing global model.

The Coverage Problem

The simplest way to handle a small device is to give it a fixed subset of the model—for example, only the first few layers or a narrow slice of every layer. However, this creates a "stale parameter" problem. If a group of devices only ever updates the first 10% of the weights, the remaining 90% of the global model never learns from their local data.

Hanhan Zhou, Tian Lan, and colleagues formalize this intuition in Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction. They introduce the minimum coverage index, a metric that tracks whether every global parameter is updated by at least one client within a certain timeframe. Their theoretical analysis proves that as long as this index is maintained—meaning the union of all local sub-models eventually spans the entire global parameter space—the heterogeneous system will converge to the same stationary point as standard, full-model FL.

Rolling Extraction and Uniform Updates

If coverage is the goal, how do we schedule it? A random selection of parameters might seem intuitive, but it often leads to high variance and "client drift," where local updates pull the model in inconsistent directions.

Samiul Alam and Mi Zhang proposed a deterministic solution called FedRolex in FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction. Instead of random sampling, the server uses a "rolling window" to extract sub-models. In each communication round, the starting index of the sub-model shifts. This ensures that every parameter is updated with approximately the same frequency over time.

This "rolling" concept is further refined for "small-to-large" scenarios—where the global model is significantly larger than any single client's capacity—by Mingfeng Xu and Chuan Chen in FedBRB. By using Block-wise Rolling, they ensure that even tiny devices contribute to every part of a massive global model in a structured sequence, preventing the formation of "untrained regions" that typically degrade performance.

Beyond Rotation: Importance-Aware Selection

While rolling windows ensure uniform coverage, they treat all parameters as equally important. However, during training, some weights contribute more to the loss reduction than others.

Zirui Xu and Xiang Chen addressed this in Helios by prioritizing "high-contribution" components—those with the largest weight changes—while rotating the remaining components to maintain long-term integrity.

Building on this, Feijie Wu and Yaqing Wang introduced FIARSE in FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extraction. FIARSE uses parameter magnitude as a proxy for importance, dynamically extracting sub-models that contain the most "relevant" parts of the evolving global model for each client. This allows hardware-constrained devices to focus their limited compute on the parameters that matter most at that specific stage of training, accelerating convergence without increasing the computational load.

Why It Works: The Collective Optimizer

The core intuition behind these methods is that Federated Learning can be viewed as a distributed optimization problem over a partitioned space. Even if no single device can see the whole model, the ensemble of devices acts as a collective optimizer.

As Yuyang Deng and Mehrdad Mahdavi explain in On the Convergence and Stability of Distributed Sub-model Training, systematic partitioning and shuffling prevent "parameter staleness." By ensuring that every weight is updated within a predictable number of rounds, the global model moves toward the optimum as if it were being trained on a single, massive machine, despite being powered by a fleet of heterogeneous, low-power devices.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_rolling_sub_model_extraction_for_heterog_328202e6,
  title = {Rolling Sub-model Extraction for Heterogeneous Federated Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rolling-sub-model-extraction-for-heterogeneous-federated-learning/rdr_328202e676b746489cf4154b28c517a3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_rolling_sub_model_extraction_for_heterog_328202e6_v1,
  title = {Rolling Sub-model Extraction for Heterogeneous Federated Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rolling-sub-model-extraction-for-heterogeneous-federated-learning/rdr_328202e676b746489cf4154b28c517a3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}