In standard Federated Learning (FL), a central server coordinates a network of edge devices to train a shared global model without directly accessing their private data. The classic baseline, Federated Averaging (FedAvg), operates on a simple assumption: each client trains a local model on its own dataset, and the server computes a weighted average of these models based purely on the number of samples each client possesses.
While this works beautifully when data is identically and independently distributed (IID), it fails dramatically in real-world scenarios where data is non-IID. Imagine a medical imaging network where one hospital specializes in rare skin conditions while another primarily sees common cases. If the server aggregates updates based solely on sample size, the massive volume of common cases will completely wash out the critical insights from the specialized hospital.
Worse yet, during local training, individual models suffer from client drift—they aggressively optimize for their local majority classes, inadvertently penalizing or "forgetting" features of classes they rarely or never see.
To solve this, a powerful paradigm has emerged: distribution-aware weighting. Instead of treating data blindly as uniform blocks of samples, this approach dynamically adjusts the influence of data at two critical stages: locally within the client's loss function, and globally during the server's aggregation step.
Local Correction: Reshaping the Client Loss Landscape
When a client's local dataset is highly skewed, its optimization landscape becomes warped. For instance, if a client only possesses images of cats, standard cross-entropy loss will continuously push the model to output "cat." In doing so, the softmax denominator actively penalizes all other classes (dogs, birds, cars), driving their logit weights downward. When this model is sent back to the server, it has effectively unlearned how to recognize anything else.
To prevent this local destruction of global knowledge, researchers modify the local objective function using class-distribution metadata.
Importance Sampling and Class Balancing
One approach is to directly scale the loss of individual samples to match a global target distribution. In Federated Visual Classification with Real-World Data Distribution, Hsu et al. introduced FedIR (Importance Reweighting), which applies importance weights to local loss functions. By evaluating local class probabilities against a global target, the local model prioritizes underrepresented classes that are globally significant without needing to see raw data from other clients.
Similarly, Luo et al. developed FedSLD in FedSLD: Federated Learning with Shared Label Distribution for Medical Image Classification. By sharing only low-risk class-wise sample counts, the algorithm scales each training sample's contribution proportionally to the ratio of the local batch distribution to the global prior, preventing skewed local datasets from hijacking the gradient direction.
Other frameworks focus on internal re-balancing. For example:
- In Knowledge-Aware Federated Active Learning with Non-IID Data, Shi et al. apply higher weights to rare classes during local training to mitigate model drift caused by skewed data.
- In Knowledge Is Not Wisdom: Weight Balancing Mechanism for Local and Global Training in Federated Learning, Kim et al. modify the loss function to emphasize underrepresented classes, balancing learning within the client's own boundaries.
- In DFML: Decentralized Federated Mutual Learning, Estiri et al. derive weights directly from the local class distribution of the current training node to prevent the model from drifting toward local majority-class biases.
- In ISFL: Federated Learning for Non-i.i.d. Data With Local Importance Sampling, Zhu et al. employ local importance sampling to dynamically adjust data point influence based on relevance to the global objective and local gradient characteristics.
Modifying the Softmax Denominator
A elegant structural fix to this problem is proposed by Legate et al. in Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning. The authors identify client drift as a form of local forgetting. They introduce Re-Weighted Softmax Cross-Entropy (WSM), which weights the contrastive term in the softmax denominator based on the local presence of classes. By zeroing out the weights for classes completely absent from a client's local dataset, the model stops penalizing labels it has not seen, preserving the global knowledge it received at the start of the round.
Global Correction: Distribution-Aware Aggregation
Fixing the local loss function is only half the battle. When local updates return to the central server, aggregating them purely by sample size ($n_i$) remains a vulnerability. If a client has a massive dataset consisting entirely of a single class, its update vector will have a huge magnitude, pulling the global model heavily toward its specific bias.
To counteract this, distribution-aware aggregation replaces sample counts with metrics that quantify how representative or skewed a client's data is relative to the global goal.
Rewarding Representativeness
In FedDisco: Federated Learning with Discrepancy-Aware Collaboration, Ye et al. argue that the server should prioritize updates from clients whose data distributions closely mirror the global target distribution. They incorporate the mathematical divergence between a client's local category distribution and the target global distribution directly into the aggregation weights. Clients with a lower discrepancy score receive higher influence, which significantly accelerates convergence and reduces communication overhead.
Statistical Distance and Skew Metrics
Instead of relying on raw category counts, models can be evaluated as statistical distributions. In FedRC: A Rapid-Converged Hierarchical Federated Learning Framework in Street Scene Semantic Understanding, Kou et al. model local datasets as Gaussian distributions and use statistical distance to calculate aggregation weights. This distribution-aware weight assignment ensures that nodes with data distributions most representative of the global target have a greater influence on the model's optimization path.
Even in non-neural architectures, distribution awareness proves vital. In FedTMOS: Efficient One-Shot Federated Learning with Tsetlin Machine, How et al. utilize a Gini index as a statistical measure of data skew to rescale client model weights during server aggregation. This ensures that models trained on dominant classes do not overshadow those trained on minority classes, maintaining global robustness across highly heterogeneous edge devices.
Why It Works: Intuition from Gradient Alignment
To understand why distribution-aware weighting works, it helps to visualize the optimization process as a collection of force vectors pulling the global model through a high-dimensional landscape.
Standard FedAvg (Unaligned Gradients):
Client A (Skewed) =======>
Client B (Balanced) -->
Global Update ========> (Heavily biased toward Client A)
Distribution-Aware Weighting (Aligned Gradients):
Client A (Reweighted) -->
Client B (Balanced) -->
Global Update ---> (Balanced, stable trajectory)
In standard FedAvg, a highly skewed client produces a massive gradient vector pointing sharply toward its local majority class. When the server averages these unaligned, highly disparate vectors, the global model takes a erratic, zigzagging path, resulting in slow convergence and poor final accuracy.
Distribution-aware weighting acts as a dual-action alignment mechanism:
- Directional Correction (Local): By altering the local loss function (e.g., via importance sampling or modifying the softmax denominator), we change the direction of the local gradient vector. It prevents the client from generating updates that destroy performance on unseen classes, ensuring the vector points closer to the true global objective.
- Magnitude Correction (Global): By altering aggregation weights based on divergence, Gini indices, or statistical distance, the server scales the magnitude of the incoming updates. Highly skewed, unrepresentative updates are dampened, while well-balanced or highly relevant updates are amplified.
By harmonizing both the direction and magnitude of client updates based on their underlying data distributions, this paradigm stabilizes the global optimization trajectory, allowing federated systems to thrive in messy, real-world environments.
Key Foundational Work
FedSLD: Federated Learning with Shared Label Distribution for Medical Image Classification (2021) — contributed 3 concept elements to this direction.
Federated Visual Classification with Real-World Data Distribution (2020) — contributed 2 concept elements to this direction.