Structure-Preserving Aggregation in Federated Learning

36 concepts18 papers1 proposal

In standard Federated Learning (FL), a central server typically aggregates local models using a simple weighted average of their parameters (FedAvg). While this works well for dense neural networks, it fails for models that rely on structural properties like sparsity or low-rankness. When different clients prune different weights to save energy or memory, their local models develop disjoint "sparsity masks." Averaging these models results in a dense global model that destroys the very efficiency and interpretability the sparsity was meant to provide.

To solve this, researchers have developed a suite of structure-preserving aggregation techniques that ensure the global model inherits the structural constraints of its local participants.

The Curse of Primal Averaging

The fundamental problem, termed the "curse of primal averaging" by Honglin Yuan et al. in Federated Composite Optimization, stems from the non-linearity of structural constraints. In optimization terms, sparsity is often enforced by a non-smooth proximal operator (like soft-thresholding for $L_1$ regularization).

Because the average of two sparse vectors with different supports is generally dense, applying the proximal operator locally before averaging at the server destroys the sparsity. As Jiaojiao Zhang et al. observe in Composite federated learning with heterogeneous data, this "primal averaging" undermines the original optimization objective, leading to global models that lack the intended structural properties.

Aggregation in the Dual Space

One powerful solution is to move the aggregation step from the primal space (model weights) to the dual space (accumulated gradients or dual multipliers).

Honglin Yuan et al. propose Federated Dual Averaging (FedDualAvg), which aggregates accumulated gradients at the server before any structural constraints are enforced. This effectively "linearizes" the communication step, allowing the server to apply a single proximal mapping to the pooled gradient information. Similarly, Site Bai and Brian Bullins demonstrate in Local Composite Saddle Point Optimization that dual-space synchronization respects the geometry of non-smooth regularizers, ensuring the global model remains compatible with sparsity or low-rank constraints. For exact sparse regression ($L_0$ norm), Tu Anh-Nguyen shows that exchanging dual multipliers allows agents to reach a consensus on a shared sparse support without ever sharing private data.

Achieving Sparsity Consensus

When models are explicitly pruned rather than regularized, the challenge shifts to "mask mismatch." If Client A decides weight $w_1$ is important and Client B decides $w_2$ is important, a simple average dilutes both signals.

To prevent divergent sparsity patterns, several frameworks enforce a sparsity consensus:

Magnitude-Preserving Merging

Even with a shared structure, the act of averaging can be problematic. If a weight is non-zero on only a subset of clients, averaging it with the zeros from other clients "dilutes" its magnitude, effectively treating a sparse signal as noise.

Guangji Bai et al. address this in FedSpaLLM: Federated Pruning of Large Language Models by proposing an aggregation method that ignores zero-valued weights. By only averaging non-zero contributions, the server preserves the signal strength of critical features identified by any participant. This element-wise logic is echoed by Peichun Li et al. in FedGreen, where only the specific weights actually updated by heterogeneous devices are averaged, ensuring that sparse updates from the edge are not washed out during global synchronization.

Key Foundational Work

Federated Composite Optimization (2021) — contributed 9 concept elements to this direction.
FedGreen: Federated Learning with Fine-Grained Gradient Compression for Green Mobile Edge Computing (2021) — contributed 1 concept elements to this direction.
Distributed High-dimensional Regression Under a Quantile Loss Function (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sparsity_preserving_aggregation_ac783f88,
  title = {sparsity-preserving aggregation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparsity-preserving-aggregation/rdr_ac783f88533048438b652b163ef40718},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sparsity_preserving_aggregation_ac783f88_v1,
  title = {sparsity-preserving aggregation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparsity-preserving-aggregation/rdr_ac783f88533048438b652b163ef40718/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake