Dual Space Aggregation for Federated Composite Optimization

This direction is merged into another direction: sparsity-preserving aggregation. Its last version stays readable and citable here.

Version 1 of 18 papers24 conceptsfrozen 2026-10-04

In standard Federated Learning (FL), the goal is to train a global model by averaging the weights of local models trained on decentralized devices. This approach, epitomized by FedAvg, works remarkably well for smooth, unconstrained problems. However, as Honglin Yuan, Manzil Zaheer, and Sashank J. Reddi point out in Federated Composite Optimization, this paradigm breaks down when we require the model to have specific structural properties, such as sparsity or low-rankness.

The Curse of Primal Averaging

The core issue lies in what researchers call the "curse of primal averaging." Imagine you want a sparse model where most parameters are zero. If Client A learns a model where only the first parameter is non-zero, and Client B learns a model where only the second parameter is non-zero, both models are sparse. However, when the server averages them, the resulting global model has two non-zero parameters.

As Jiaojiao Zhang and Mikael Johansson explain in Composite federated learning with heterogeneous data, the average of multiple sparse vectors with different support sets is typically dense. This means that even if every client uses a non-smooth regularizer (like the L1 norm) to induce sparsity locally, the server-side averaging step destroys that sparsity. The global model loses the very structural properties the optimization objective was designed to produce.

Aggregating in the Dual Space

To solve this, a new family of algorithms shifts the point of consensus from the "primal" space (the model weights) to the "dual" space (the accumulated gradients or dual variables).

In Federated Composite Optimization, the authors introduce Federated Dual Averaging (FedDualAvg). Instead of averaging weights, the server aggregates the gradient information sent by clients. The key intuition is that the structural constraints—enforced by a mathematical tool called a "proximal operator"—should be applied after the global information has been pooled, not before.

By maintaining a global dual variable that tracks the sum of all client gradients, the server can apply the proximal operator to the aggregated information. This ensures that the global model remains sparse or low-rank because the "sparsifying" step happens on the combined knowledge of the entire network, rather than being diluted by the averaging of pre-sparsified weights.

Preserving Structure in Complex Problems

This principle of dual-space aggregation is not limited to simple minimization. In Local Composite Saddle Point Optimization, Site Bai and Brian Bullins extend this logic to composite saddle point problems (SPPs). These problems are common in adversarial training and games where one must balance competing objectives.

Their algorithm, Federated Dual Extrapolation (FeDualEx), performs synchronization in a representation space that respects the geometry of non-smooth regularizers. By performing dual extrapolation—a technique that uses gradient information from a "look-ahead" point—and aggregating those updates in the dual space, they maintain model structure even in the more volatile landscape of minimax optimization.

Why It Works: A Change in Perspective

The shift from primal to dual aggregation works because it changes the order of operations:

  1. Standard FL (FedAvg): Local Proximal Step $\rightarrow$ Primal Averaging (Destroys Structure).
  2. Dual FL (FedDualAvg/FeDualEx): Dual Aggregation $\rightarrow$ Global Proximal Step (Preserves Structure).

By delaying the application of the non-smooth mapping until after the clients' contributions are merged, these methods ensure that the global model inherits the intended constraints. As Jiaojiao Zhang notes, aggregating gradient-related states rather than final local models prevents the "density issues" that plague traditional federated averaging, making FL viable for high-dimensional tasks where interpretability and efficiency are paramount.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dual_space_aggregation_for_federated_com_c5fce5a3,
  title = {Dual Space Aggregation for Federated Composite Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-space-aggregation-for-federated-composite-optimization/rdr_c5fce5a3d50a41e88dc9172e6ff4579e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dual_space_aggregation_for_federated_com_c5fce5a3_v1,
  title = {Dual Space Aggregation for Federated Composite Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-space-aggregation-for-federated-composite-optimization/rdr_c5fce5a3d50a41e88dc9172e6ff4579e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}