In standard Federated Learning (FL), the central server typically treats every client update as an equal vote, or weights them simply by the number of samples a client holds. This works well when data is "Independent and Identically Distributed" (IID). However, in the real world—across hospitals, satellites, or mobile devices—data is rarely uniform. One hospital might see thousands of common flu cases but only two instances of a rare disease, while another hospital sees the opposite.
When data is non-IID, standard averaging causes the global model to "drift" toward the majority classes of the most data-heavy clients. Recent research has converged on a shared solution: Distribution-Aware Weighting. By adjusting weights at both the local training level and the global aggregation level, researchers are ensuring that rare but critical information isn't drowned out by redundant, common data.
Correcting Local Bias with Importance Weighting
The first line of defense against data skew happens on the client device itself. If a client’s local dataset is heavily biased, the model it trains will naturally become "lazy," over-optimizing for the majority class.
To combat this, Jun Luo and Shandong Wu proposed FedSLD, which re-weights the local loss function based on the ratio of local class counts to the global distribution. Intuitively, if a client knows that a specific class is rare globally but common locally, it can down-weight those samples to prevent its update from "hijacking" the global gradient.
Similarly, Tzu-Ming Harry Hsu and colleagues introduced FedIR (Importance Reweighting) in Federated Visual Classification with Real-World Data Distribution. This approach aligns heterogeneous client distributions with a global target distribution. By applying these importance weights, local models prioritize underrepresented classes that are globally significant, allowing federated models to reach accuracy levels nearly identical to centralized training.
Prioritizing Diversity during Aggregation
Even if local training is balanced, the server still faces a challenge: how much should it trust each client's update? Simple averaging fails when some clients contribute highly redundant information while others contribute unique, high-variance insights.
Researchers are now using statistical measures of "diversity" to decide aggregation weights:
- Statistical Skew: Shannon How and the team behind FedTMOS use the Gini index to measure data skew. They rescale client weights so that models trained on dominant classes do not overshadow those trained on minority classes.
- Information Uniqueness: In decentralized vehicle networks, Xiaoyan Liu developed C-DFL, which weights updates based on the volume of unique, non-redundant data. This prevents the model from stalling when multiple sensors report the same redundant environment.
- Projection as Proxy: In aerial networks, the WeiAvgCS framework uses a "projection" metric as a privacy-preserving proxy for data diversity. By prioritizing updates from high-diversity clients, they achieved up to 46% faster convergence.
From Similarity to Affinity
While most FL methods try to group similar clients, a new direction suggests that the best way to solve imbalance is to seek out opposites.
Yang Xu and Ye Ding proposed DA-PFL, which uses Dynamic Affinity Aggregation. Instead of aggregating models from clients with similar data, it groups clients with complementary distributions. If Client A has only "Class 1" and Client B has only "Class 2," they have high affinity. By merging these complementary biases, the global model fills in its own knowledge gaps more efficiently than it would by simply averaging similar, biased models.
Why It Works: The Intuition of Balance
The shared insight across these 20 papers is that data volume is a poor proxy for data value. In a non-IID world, a small dataset containing a rare disease is often more "valuable" to the global model's generalization than a massive dataset of common cases.
By using distribution-aware weighting—whether through cost-sensitive loss functions as surveyed by Chuanwen Li or through diversity-aware aggregation—the federated system transforms from a simple "average of parts" into a sophisticated "synthesis of expertise." This ensures that the final model reflects the true global prevalence of data rather than the loudest local voices.