In centralized deep learning, Batch Normalization (BN) is a foundational component that stabilizes and accelerates training. It operates on a simple assumption: a mini-batch of data sampled during training serves as a reliable proxy for the global data distribution. By computing the mean and variance of this mini-batch, BN standardizes intermediate activations, smoothing the loss landscape.
In decentralized and federated learning, however, this fundamental assumption collapses. When data is distributed across heterogeneous clients in a non-identically distributed (non-IID) fashion, local mini-batches reflect only local biases rather than the global population. This statistical discrepancy introduces severe training instability, weight divergence, and dramatic accuracy degradation.
The Core Mechanism of Failure
To understand why batch normalization fails in decentralized environments, we must look at what happens during local training and subsequent model aggregation.
When a client trains a local model on its unique data slice, its BN layers compute running estimates of mean and variance that are heavily biased toward that client's specific distribution. As identified by Hanlin Tang et al. in D2: Decentralized Training over Decentralized Data, these local statistics fail to align with global model requirements. When a central server aggregates these local models, it averages parameters that were optimized under completely different feature-scaling regimes.
This mismatch acts as a form of internal covariate shift that poisons the global model. Amar Phanishayee et al. demonstrate in The Non-IID Data Quagmire of Decentralized Machine Learning that aggregating these mismatched local statistics directly degrades the global model's performance, turning standard normalization layers into a primary culprit for accuracy loss.
Centralized Training (IID):
Global Data ──> Uniform Mini-Batch ──> Accurate Global Stats (μ, σ) ──> Stable Training
Federated Training (Non-IID):
Client A Data ──> Local Mini-Batch ──> Biased Stats A (μ_A, σ_A) ──┐
├──> Aggregation Collapse
Client B Data ──> Local Mini-Batch ──> Biased Stats B (μ_B, σ_B) ──┘
The Dual Mismatch: Forward and Backward Passes
The instability induced by BN is not merely a consequence of divergent forward-pass statistics; it is a dual-pass failure. In Why Batch Normalization Damage Federated Learning on Non-IID Data?, Qingjiang Shi et al. provide a rigorous breakdown showing that BN suffers from a dual mismatch involving both forward-pass statistical parameters and backward-pass gradients.
Because the activation scaling is inconsistent across clients, the gradient landscapes computed during the backward pass become highly contradictory. Synchronizing the means and variances alone cannot fix this issue if the underlying gradient trajectories remain misaligned. This phenomenon causes local model updates to drift significantly from the global objective, leading to severe weight divergence during aggregation (Sara Pieri et al. in Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition).
This problem is further compounded by system-level constraints. When partial client participation is combined with frequent parameter averaging across heterogeneous nodes, BN layers are prevented from ever stabilizing, resulting in highly divergent updates and poor convergence (Qinbin Li et al. in Federated Learning on Non-IID Data Silos: An Experimental Study; Jingyu Pan et al. in Towards Collaborative Intelligence: Routability Estimation based on Decentralized Private Data).
Architectural Remedies: Eliminating Batch Dependency
Because the root cause of the instability is BN's dependency on batch statistics, a prominent line of research focuses on architectural modifications that eliminate this dependency entirely.
Group and Layer Normalization
Replacing BN with Group Normalization (GN) or Layer Normalization (LN) has proven highly effective. As highlighted by Hangyu Zhu et al. in Federated Learning on Non-IID Data: A Survey and Khalil Jahani et al. in A Survey on Data Distribution Challenges and Solutions in Vertical and Horizontal Federated Learning, GN operates on channels within a single sample rather than across a batch. This makes it entirely independent of batch size and highly robust against data heterogeneity across different nodes.
Similarly, the authors of Quasi-global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data note that substituting BN with statistics-independent layers mitigates local bias and stabilizes convergence. Sara Pieri et al. demonstrate that swapping BN for LN stabilizes weight divergence and improves global model performance without requiring complex optimization adjustments.
Feature Normalization
To understand why alternative normalization layers succeed, Guojun Zhang et al. isolated the mechanism in Understanding the Role of Layer Normalization in Label-Skewed Federated Learning. They discovered that the primary benefit comes from Feature Normalization (FN) applied right before the classifier head. Under extreme label shift, FN prevents feature collapse and local overfitting, addressing unique failure modes of decentralized optimization rather than merely changing the network's expressive power.
Removing Normalization Layers
Taking this insight to its logical conclusion, Weiming Zhuang et al. developed a framework in Is Normalization Indispensable for Multi-domain Federated Learning? that removes normalization layers entirely. By replacing them with Scaled Weight Standardization, they completely eliminate the dependency on local data statistics, allowing models to learn domain-agnostic features across highly diverse data silos.
Algorithmic Remedies: Localizing and Freezing Statistics
An alternative strategy involves retaining BN layers for their optimization benefits but altering how their parameters are aggregated and updated.
Localizing Normalization Layers
Instead of forcing a single set of normalization statistics onto all clients, personalized federated learning frameworks isolate BN layers. In Personalized Federated Learning via Backbone Self-Distillation, Pengju Wang et al. demonstrate that keeping normalization layers local while sharing the rest of the network's backbone parameters mitigates feature representation bias.
This concept is expanded by Ligang He et al. in PFed-NS: An Adaptive Personalized Federated Learning Scheme Through Neural Network Segmentation, who show that maintaining local BN layers during periods of high divergence prevents performance degradation. Transitioning to global aggregation only after training stabilizes ensures that the global model does not disrupt specialized local feature distributions.
Freezing Statistics
We can also stabilize BN by controlling when its statistics are allowed to change. Jike Zhong et al. challenge the assumption that BN should always be discarded in Making Batch Normalization Great in Federated Deep Learning. They observe that BN can actually outperform GN when clients perform multiple local updates between communication rounds. To exploit this, they introduce a strategy that freezes BN statistics during specific training stages, successfully bridging the performance gap between decentralized and centralized learning without abandoning the architecture of standard convolutional networks.
Key Foundational Work
The Non-IID Data Quagmire of Decentralized Machine Learning (2020) — contributed 2 concept elements to this direction.
D2: Decentralized Training over Decentralized Data (2018) — contributed 2 concept elements to this direction.