Abstract
Federated Learning (FL) enables collaborative model training without sharing raw data, yet its performance is frequently hampered by non-independent and identically distributed (non-i.i.d.) data across participants. We present Feature Norm Regularized Federated Learning (FNR-FL), a framework that utilizes class-average feature norms as a privacy-preserving regularization tool to mitigate model drift. By aligning internal feature representations across local models, FNR-FL achieves substantial accuracy improvements—up to 66% over standard FedAvg—while simultaneously enhancing training efficiency.
The Challenge of Non-i.i.d. Data in Federated Learning
The primary obstacle in Federated Learning is the diversity of local data distributions, commonly referred to as non-i.i.d. data. In real-world deployments, participants may exhibit various forms of data heterogeneity:
- Label Skew: Participants hold different proportions of classes (e.g., one user has only 'cat' images).
- Quantity Skew: Some participants possess significantly more data than others.
- Feature Distribution Skew: Data quality varies, such as differences in image clarity or noise levels.
When standard algorithms like FedAvg aggregate these divergent local models, the resulting global model often suffers from poor convergence and low accuracy. FNR-FL aims to bridge these gaps by regularizing the local training process to ensure updates are more compatible.
Methodology: FNR-FL
FNR-FL introduces a "plug-and-play" regularization module designed to stabilize local updates. The process involves four key steps:
- Feature Norm Calculation: Local models compute the $L_2$ norm of features extracted just before the final classification layer for each sample.
- Class Averaging: These norms are averaged per class to create a compact representation of the local feature distribution.
- Targeted Regularization: The central server identifies "struggling" participants—those with low accuracy on a small public proxy dataset—to apply additional regularization.
- Loss Function Enhancement: A penalty term is added to the local cross-entropy loss, discouraging deviations between the local class feature norms and the global averages.
This approach forces local models to align their internal representations with the collective knowledge of the federation, preventing individual models from drifting due to local data biases.
Experimental Setup and Datasets
The framework was evaluated on three benchmarks: CIFAR-10, MNIST, and FEMNIST. To simulate realistic conditions, the researchers implemented mixed non-i.i.d. scenarios combining multiple types of skew.
By introducing varying levels of noise, the study demonstrated how FNR-FL handles feature distribution shifts that typically degrade standard FL performance. The variance in feature norms serves as an effective proxy for these distribution differences without compromising raw data privacy.
Key Results and Efficiency Gains
FNR-FL demonstrates superior performance across diverse non-i.i.d. settings:
- Accuracy Improvements: On CIFAR-10 with feature skew, FNR-FL reached 99.76% accuracy, significantly outperforming FedAvg's 60.01%.
- Robustness in Mixed Scenarios: In complex "label + quantity skew" environments, FNR-FL maintained 88.26% accuracy, while competing methods like FedProx and SCAFFOLD dropped to approximately 30%.
- Training Efficiency: FNR-FL achieved a 66.24% boost in accuracy alongside an 11.40% reduction in training time compared to FedAvg, proving that better alignment leads to faster convergence.
- Compatibility: The method is "orthogonal," meaning it can be integrated with other FL optimizations (e.g., MOON, FedNova) for cumulative benefits.
Conclusion and Future Directions
FNR-FL effectively uses feature norms as a proxy for data distribution differences, leading to more stable and accurate global models. While the current implementation requires a small public dataset for evaluation, future work will explore removing this dependency and further optimizing communication overhead. Additionally, integrating Differential Privacy (DP) could provide stronger formal privacy guarantees against potential inference attacks.
Summary updated