Global Flatness in Heterogeneous Federated Learning

51 concepts23 papers1 proposal

In machine learning, the geometry of the loss landscape often dictates how well a model generalizes. Models that land in "flat minima"—broad regions where the loss changes slowly with respect to parameter perturbations—tend to perform better on unseen data than those in "sharp minima."

In Federated Learning (FL), where models are trained across decentralized devices with heterogeneous data, achieving this flatness is notoriously difficult. Recent research has identified a fundamental local-global flatness discrepancy: even when individual clients successfully find flat regions on their local data, the aggregated global model often ends up in a sharp, poorly generalizing region of the global landscape.

The Failure of Local Sharpness

To improve generalization, many FL algorithms incorporate Sharpness-Aware Minimization (SAM), a technique that explicitly seeks out flat regions by adding perturbations to model weights during training. However, Lee et al. and Liu et al. demonstrate that local flatness is not a reliable proxy for global generalization.

The core issue is data heterogeneity. As Li et al. observe, diverse data distributions across clients do not just shift model parameters; they actively warp the underlying loss landscape. A region that appears as a broad, flat basin for one client’s data might correspond to a steep, sharp peak when evaluated against the combined data of all participants. Liu et al. quantify this as the "flatness distance," showing that as data heterogeneity increases, the gap between local and global flatness widens, leading to significant performance degradation.

Why Aggregation Sharpens the Landscape

When a central server averages models from different clients, it assumes that the local updates are moving toward a shared, stable equilibrium. But if local objectives are inconsistent, their average may reside in an unstable region. Sun et al. and others in Sun et al. argue that improving the smoothness of individual local landscapes does not inherently ensure the smoothness of the aggregated global landscape.

This misalignment occurs because:

  1. Perturbation Drift: A perturbation direction that seems "sharp" to a local client may be irrelevant to the global model's geometry (Fan et al.).
  2. Conflicting Updates: Heterogeneity causes nodes to produce conflicting update vectors, preventing the global model from converging to the flat minima essential for robust generalization (Yashwanth et al.).
  3. Curvature Mismatch: Simple averaging fails to account for second-order heterogeneity—the differences in how the loss landscape curves across different clients (Murata et al.).

Seeking Global Awareness

To bridge this gap, recent methods shift the focus from local tweaks to global awareness. The goal is to ensure that the aggregated model—not just the local ones—lands in a flat minimum.

Global Trajectories as Proxies Since clients cannot see the full global dataset, they must estimate the global landscape using available information. Fan et al. (FedLESAM) and Li et al. (FedGMT) propose using the historical trajectory of the global model to estimate global perturbation directions. By aligning local updates with the global model's path, clients can identify flat regions that are consistent across the entire network.

Synchronized Regularization Other approaches use regularization to force alignment. Lee et al. (FedGF) adaptively interpolate between local and global perturbations. Similarly, Sun et al. (FedSMOO) combine dynamic regularization with a globally-synchronized SAM objective to ensure that local updates do not drift away from a shared global trajectory.

Server-Side Optimization Rather than relying solely on client-side adjustments, Caldarola et al. (FedGloSS) suggest optimizing for global sharpness at the server level. By reusing previous gradients to approximate the global landscape's curvature, the server can guide the model toward stable regions without increasing the communication overhead.

The Path to Robust Generalization

The shift from local to global flatness optimization represents a maturing understanding of decentralized learning. As Yan et al. argue, robust generalization requires a dual focus: local flatness to prevent overfitting to individual client data, and global flatness to ensure stability across the entire federated network. By resolving the local-global discrepancy, these methods allow federated models to achieve the high-performance generalization typically reserved for centralized training.

Key Foundational Work

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning (2022) — contributed 1 concept elements to this direction.
FedShuffle: Recipes for Better Use of Local Work in Federated Learning (2022) — contributed 1 concept elements to this direction.
Federated Self-supervised Learning for Video Understanding (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_local_global_flatness_discrepancy_5abddee8,
  title = {local-global flatness discrepancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-global-flatness-discrepancy/rdr_5abddee8c7894745b5aafa6d75ddbf33},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_local_global_flatness_discrepancy_5abddee8_v1,
  title = {local-global flatness discrepancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-global-flatness-discrepancy/rdr_5abddee8c7894745b5aafa6d75ddbf33/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake