In the ideal version of Federated Learning (FL), a stable set of clients—such as hospitals or banks—collaboratively trains a model by communicating with a central server over many rounds. In this "cross-silo" setting, clients are persistent; they can store local model weights, momentum buffers, and optimizer states between rounds.
However, as Ekdeep Singh Lubana et al. argue in Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering, the reality of "cross-device" FL is much messier. When training on millions of smartphones or IoT sensors, devices participate sporadically, often connecting only once during the entire training process. This creates a statelessness constraint: the system cannot assume a client will be available to retrieve or update a stored local state in the future.
Research in this area focuses on bridging the gap between the high-performance algorithms used in stable environments and the volatile, stateless reality of the edge.
The Problem with Memory
Most advanced optimization techniques are inherently stateful. For example, adaptive optimizers like Adam or Adagrad require tracking moving averages of gradients. As Sashank J. Reddi et al. point out in Adaptive Federated Optimization, requiring clients to maintain these buffers is infeasible when participation is unpredictable. If a client drops out, its local "memory" of the optimization trajectory is lost, leading to what is known as client drift.
This lack of continuity also hampers personalization. Traditionally, personalizing a model meant a device would keep a small part of the model private and update it over time. Tian Li et al. note in Ditto: Fair and Robust Federated Learning Through Personalization that if a device cannot store this local state, the benefits of personalization are lost to new or infrequent participants.
Mimicking State through Global Information
To solve this, researchers have developed methods that substitute local history with global information. Instead of asking a client to remember its own past, the server provides a "summary" of the collective progress.
In Breaking the centralized barrier for cross-device federated learning, Martin Jaggi et al. introduce MIME, which forces local updates to mimic centralized algorithms. By using global optimizer states transmitted from the server, MIME allows stateless clients to achieve the same drift-correction benefits as stateful ones. Similarly, Geeho Kim et al. propose FedACG, which uses global momentum as a "lookahead" mechanism. This allows the global model to act as a proxy for the local history that stateless devices lack.
Reconstruction instead of Storage
If a device cannot store a personalized model, can it "re-discover" it on the fly? This is the intuition behind Federated Reconstruction. Karan Singhal et al. treat personalization as a fast "reconstruction" task. When a stateless client joins a round, it briefly trains a set of local parameters from scratch before performing the actual global update. This meta-learning approach allows for massive-scale personalization without requiring any device to store state between rounds.
A similar logic is applied by Konstantin Mishchenko et al. in Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity. By re-initializing local parameters at the start of every communication round, the authors prove that models can still converge efficiently even when clients are entirely transient.
Removing Stateful Layers
Sometimes, the model architecture itself is the bottleneck. Standard neural networks rely on Batch Normalization, which tracks running statistics of data—a form of local state. In Is Normalization Indispensable for Multi-domain Federated Learning?, Weiming Zhuang et al. demonstrate that removing these stateful layers in favor of Weight Standardization allows models to generalize better across diverse, stateless clients.
By shifting the burden of "memory" from the volatile edge to the stable server, or by enabling devices to rapidly reconstruct their context, these researchers are making federated learning viable for the billions of devices that make up the modern mobile ecosystem.
Key Foundational Work
Divergence-aware Federated Self-Supervised Learning (2022) — contributed 2 concept elements to this direction.
Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering (2022) — contributed 2 concept elements to this direction.
Federated Reconstruction: Partially Local Federated Learning (2021) — contributed 1 concept elements to this direction.