The Convergence Gap in Advanced Federated Learning

55 concepts56 papers1 proposal

Federated learning (FL) promises a compelling vision: training a global machine learning model across decentralized edge devices while keeping data strictly private. In its simplest form, algorithms like Federated Averaging (FedAvg) operate under clean mathematical assumptions—such as independent and identically distributed (IID) data, uniform client participation, and synchronized updates. Under these idealized conditions, optimization theorists can rigorously prove that the global model will steadily march toward an optimal solution.

However, real-world deployments rarely match these clean assumptions. Edge devices suffer from severe data heterogeneity (non-IID data), varying hardware capabilities, and unpredictable network connections. To survive in the wild, the field has seen an explosion of highly creative, empirical heuristics: personalized sub-networks, dynamic client routing, adaptive learning rates, and aggressive compression schemes.

This has created a profound theoretical convergence gap. While these advanced frameworks achieve remarkable empirical success, they often lack formal mathematical proofs ensuring they will reliably converge. As a scoping review by Siqi Li et al. highlights, there is a notable absence of formal convergence analysis across many newly proposed federated learning frameworks. This leaves it unclear whether these decentralized algorithms can reliably reach optimal parameters when applied to complex, heterogeneous real-world datasets.

Why Advanced Heuristics Break Traditional Proofs

To understand why theory lags behind practice, we must look at what happens mathematically when we modify the standard federated optimization loop. A classic convergence proof relies on tracking a single global parameter vector moving across a well-behaved loss landscape. Advanced FL algorithms break this paradigm in three fundamental ways.

1. Personalization and Split Architectures

Instead of forcing every client to learn the exact same model, personalized FL (pFL) allows clients to maintain local variations to combat data heterogeneity. For instance, Hamed Hassani et al. introduced FedRep, which splits models into shared global representations and unique local heads. Similarly, Eric Lei et al. proposed Fed-NTC, which shares global transforms but personalizes local entropy models for image compression.

When a model's architecture is split or dynamically routed—such as the sample-specific routing policy proposed by Jianqing Zhang et al. in FedCP—the optimization landscape becomes a moving target. Because clients are optimizing different local objectives or routing features through different sub-networks, the global aggregation step no longer represents a clean average of gradients on a single shared surface. Consequently, proving that these hybrid update schemes will reach a stable equilibrium remains an open challenge.

2. Dynamic Routing and System Adaptivity

In real-world networks, a server cannot assume all clients will respond synchronously or possess identical hardware. Frameworks like FedSAE, developed by Duo Liu et al., dynamically adjust local training workloads based on historical completion rates to mitigate stragglers. Other approaches, like the adaptive learning rate mechanisms studied by Wenkai Yang et al., adjust optimization steps on the fly to handle negative client sampling.

When the optimization parameters (like workloads, participation probabilities, or learning rates) depend dynamically on the state of the system, the standard stochastic gradient descent (SGD) assumptions break down. For example, Angelo Rodio et al. note that while convergence can be proven under simplified independent Bernoulli participation distributions, extending these proofs to realistic, complex client dynamics remains an open question. Without formal proofs, it is difficult to guarantee that a self-adaptive system won't enter an unstable feedback loop.

3. Aggressive Compression and Representation Alignment

To minimize massive communication overhead, researchers use advanced mathematical transformations. Yo-Seb Jeon et al. leverage compressive sensing to reconstruct sparse local gradients, while Anh Van Nguyen et al. utilize Tucker tensor decomposition for multimodal image reconstruction.

Other frameworks attempt to align entirely different model architectures across clients. FedHeNN, proposed by Disha Makhija et al., uses similarity metrics like Centered Kernel Alignment (CKA) to align instance-level representations across heterogeneous client networks.

While these methods drastically reduce communication costs or allow architectural freedom, they introduce non-linearities and approximations that defy standard convergence analysis. Proving that a global model will consistently reach an optimum when updates are heavily compressed, sketched, or aligned via abstract similarity metrics requires entirely new mathematical frameworks that have yet to be fully realized.

The Pervasiveness of Unproven Convergence

This theoretical gap is not isolated to a single niche; it spans nearly every frontier of modern federated learning research:

  • Unsupervised Learning: While supervised FL has foundational theory, unsupervised tasks like decentralized clustering lack rigorous performance guarantees, as noted by Ye Tian et al..
  • Clustered and Sequential FL: Grouping clients into clusters based on final-layer weights (Md Sirajul Islam et al.) or passing a model sequentially from client to client like a relay race (Yipeng Li et al.) show massive empirical promise but frequently lack the comprehensive mathematical frameworks needed to guarantee long-term stability.
  • Multi-Epoch Training: Even basic adjustments, like allowing clients to train for multiple local epochs rather than a single one, introduce mathematical complexities that make global model stability difficult to prove formally (Minsu Kim et al.).
  • Fine-Tuning Foundation Models: Modern paradigms like HETLORA (Matt Barnes et al.), which allows edge devices to train using different low-rank adapter ranks based on hardware constraints, currently operate without formal convergence guarantees.

Why Closing the Gap Matters

Why should practitioners care about the lack of a mathematical proof if the empirical curves look good?

In centralized deep learning, empirical success is often sufficient because the training environment is fully controlled. In federated learning, however, the training environment is inherently chaotic, distributed, and adversarial. Without rigorous convergence guarantees, an algorithm that performs beautifully on standard benchmarks might fail catastrophically when deployed across a real-world network of edge devices.

Unexpected client dropouts, extreme data shifts, or modular arithmetic wrap-around events (as explored by Brendan McMahan et al.) can trigger divergence, client drift, or model degradation. Bridging this gap requires optimization theorists and empirical researchers to work in tandem—building new mathematical tools that can accommodate the messy, adaptive, and personalized realities of modern decentralized learning.

Key Foundational Work

On Safeguarding Privacy and Security in the Framework of Federated Learning (2020) — contributed 1 concept elements to this direction.
Federated Learning with Autotuned Communication-Efficient Secure Aggregation (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lack_of_theoretical_guarantees_e920b61c,
  title = {lack of theoretical guarantees},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lack-of-theoretical-guarantees/rdr_e920b61c482b4bcf85ce84195d1794c1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_lack_of_theoretical_guarantees_e920b61c_v1,
  title = {lack of theoretical guarantees},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lack-of-theoretical-guarantees/rdr_e920b61c482b4bcf85ce84195d1794c1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake