In standard Federated Learning (FL), a central server typically aggregates local models using a simple weighted average of their parameters (FedAvg). While this works well for dense neural networks, it fails for models that rely on structural properties like sparsity or low-rankness. When different clients prune different weights to save energy or memory, their local models develop disjoint "sparsity masks." Averaging these models results in a dense global model that destroys the very efficiency and interpretability the sparsity was meant to provide.
To solve this, researchers have developed a suite of structure-preserving aggregation techniques that ensure the global model inherits the structural constraints of its local participants.
The Curse of Primal Averaging
The fundamental problem, termed the "curse of primal averaging" by Honglin Yuan et al. in Federated Composite Optimization, stems from the non-linearity of structural constraints. In optimization terms, sparsity is often enforced by a non-smooth proximal operator (like soft-thresholding for $L_1$ regularization).
Because the average of two sparse vectors with different supports is generally dense, applying the proximal operator locally before averaging at the server destroys the sparsity. As Jiaojiao Zhang et al. observe in Composite federated learning with heterogeneous data, this "primal averaging" undermines the original optimization objective, leading to global models that lack the intended structural properties.
Aggregation in the Dual Space
One powerful solution is to move the aggregation step from the primal space (model weights) to the dual space (accumulated gradients or dual multipliers).
Honglin Yuan et al. propose Federated Dual Averaging (FedDualAvg), which aggregates accumulated gradients at the server before any structural constraints are enforced. This effectively "linearizes" the communication step, allowing the server to apply a single proximal mapping to the pooled gradient information. Similarly, Site Bai and Brian Bullins demonstrate in Local Composite Saddle Point Optimization that dual-space synchronization respects the geometry of non-smooth regularizers, ensuring the global model remains compatible with sparsity or low-rank constraints. For exact sparse regression ($L_0$ norm), Tu Anh-Nguyen shows that exchanging dual multipliers allows agents to reach a consensus on a shared sparse support without ever sharing private data.
Achieving Sparsity Consensus
When models are explicitly pruned rather than regularized, the challenge shifts to "mask mismatch." If Client A decides weight $w_1$ is important and Client B decides $w_2$ is important, a simple average dilutes both signals.
To prevent divergent sparsity patterns, several frameworks enforce a sparsity consensus:
- Shared Masks: Sixu Hu et al. argue in Practical Hybrid Gradient Compression for Federated Learning Systems that using a shared sparsification mask ensures the global update remains as sparse as the local ones, which is critical for efficient downlink compression.
- Server-Led Pruning: Sara Babakniya et al. propose periodic sub-sampling at the server to prune the global model back to a unified structure. Lei Li et al. extend this in An Efficient and Accurate Dynamic Sparse Training Framework Based on Parameter-Freezing by centralizing mask management to filter out noise from non-IID client data.
- Hard Thresholding: In Federated Gradient Matching Pursuit, Deanna Needell et al. apply a hard-thresholding operation after aggregation to consolidate feature importance across the fleet.
Magnitude-Preserving Merging
Even with a shared structure, the act of averaging can be problematic. If a weight is non-zero on only a subset of clients, averaging it with the zeros from other clients "dilutes" its magnitude, effectively treating a sparse signal as noise.
Guangji Bai et al. address this in FedSpaLLM: Federated Pruning of Large Language Models by proposing an aggregation method that ignores zero-valued weights. By only averaging non-zero contributions, the server preserves the signal strength of critical features identified by any participant. This element-wise logic is echoed by Peichun Li et al. in FedGreen, where only the specific weights actually updated by heterogeneous devices are averaged, ensuring that sparse updates from the edge are not washed out during global synchronization.
Key Foundational Work
Federated Composite Optimization (2021) — contributed 9 concept elements to this direction.
FedGreen: Federated Learning with Fine-Grained Gradient Compression for Green Mobile Edge Computing (2021) — contributed 1 concept elements to this direction.
Distributed High-dimensional Regression Under a Quantile Loss Function (2019) — contributed 1 concept elements to this direction.