In centralized deep learning, Batch Normalization (BN) is a nearly universal tool for stabilizing training and accelerating convergence. However, when the training process is distributed across a network of decentralized devices—a paradigm known as Federated Learning (FL)—BN often becomes a liability rather than an asset.
The core of the problem lies in a mismatch between local data and global goals. In the real world, data is rarely "IID" (Independent and Identically Distributed). As Kevin Hsieh and colleagues describe in The Non-IID Data Quagmire of Decentralized Machine Learning, a camera in Australia might see only kangaroos while a camera in Norway sees only lynxes. This "label skew" creates a statistical divergence that standard normalization layers cannot handle.
The Failure of Local Statistics
To understand why BN fails, we must look at how it operates. BN standardizes activations using the mean and variance of the current minibatch. In a centralized setting, a minibatch is a representative slice of the entire dataset. In a decentralized setting, the minibatch reflects only the local, skewed distribution of a single client.
When clients normalize their data using these biased local statistics, they inadvertently perform an "external covariate shift." As Rachid Guerraoui and John Stephan point out in Overcoming the Challenges of Batch Normalization in Federated Learning, this process can map distinct classes from different clients into overlapping feature spaces. If every client forces their unique data into the same zero-mean, unit-variance distribution, the global model loses the ability to tell those classes apart.
Furthermore, Qingjiang Shi and Tsung-Hui Chang have shown that the damage is two-fold: not only are the forward-pass statistics (mean and variance) inconsistent across clients, but the backward-pass gradients are also mismatched. This dual mismatch biases global updates, preventing the model from ever reaching an optimal solution.
Architectural Solutions: Moving Beyond the Batch
One of the most effective ways to solve the normalization quagmire is to change the architecture itself. Since the problem stems from a dependency on batch statistics, researchers have proposed using normalization techniques that operate on individual samples or groups of channels.
- Group and Layer Normalization: Kevin Hsieh and Sara Pieri demonstrate that replacing BN with Group Normalization (GN) or Layer Normalization (LN) can restore model accuracy. Because these methods do not rely on minibatch statistics, they are inherently robust to the label skew found in decentralized environments.
- Normalization-Free Networks: In an even more radical shift, Weiming Zhuang and Lingjuan Lyu introduced FedWon, a framework that removes normalization layers entirely. By using Scaled Weight Standardization instead, they allow models to learn domain-agnostic features without the instability introduced by divergent local statistics.
Fixing Batch Normalization
Despite its flaws, BN remains popular because of its performance benefits in centralized settings. Consequently, a significant body of research focuses on "fixing" BN for federated contexts rather than discarding it.
In Making Batch Normalization Great in Federated Deep Learning, Jike Zhong and Hong-You Chen propose FIXBN. This strategy involves a two-stage training process where BN statistics are eventually frozen, bridging the gap between decentralized and centralized performance. Similarly, Rachid Guerraoui introduced Federated BatchNorm (FBN), which uses shared global statistics and a mathematical correction to eliminate the bias in variance aggregation.
Optimization as a Stabilizer
Beyond architectural changes, the stability of normalization is closely tied to the optimization algorithm used. Standard decentralized SGD often leads to "client drift," where local models diverge too far from the global objective.
Algorithms like $D^2$, developed by Hanlin Tang and Ji Liu 0002, use variance reduction to ensure that workers can train locally for longer periods without drifting apart. In the same vein, Tao Lin and Sebastian U Stich introduced Quasi-Global (QG) Momentum. By using the difference between synchronized models to approximate a global direction, QG Momentum helps stabilize training even when local normalization layers are providing noisy, biased signals.
Summary of the Direction
The research trajectory across these 52 papers suggests a fundamental shift in how we view distributed training. We are moving away from treating Federated Learning as a pure optimization problem and toward a more holistic "federated-aware" architectural design. Whether by replacing Batch Normalization with more robust alternatives or by carefully synchronizing global statistics, the goal is to ensure that the internal mechanics of a neural network can withstand the messy, non-IID reality of decentralized data.