Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains

Ke Hu, Liyao Xiang, Peng Tang, Weidong Qiu

Abstract

Federated Learning (FL) enables collaborative model training without sharing raw data, but its performance is severely degraded by non-i.i.d. data distributions across participants. We propose Feature Norm Regularized Federated Learning (FNR-FL), a novel approach that utilizes class-average feature norms as a proxy for data disparity. By selectively regularizing participants with low accuracy to align their feature distributions, FNR-FL mitigates model drift and improves convergence. Our evaluation on CIFAR-10, MNIST, and FEMNIST demonstrates that FNR-FL significantly outperforms state-of-the-art methods, achieving up to a 66% accuracy improvement over FedAvg while reducing training time by 11.4%.

Introduction

Federated Learning (FL) is a promising paradigm for privacy-preserving machine learning, enabling multiple parties to train a shared model without exchanging raw data. However, the real-world application of FL is hindered by the non-i.i.d. (non-independent and identically distributed) data problem. Variations in data quantity, label distributions, and feature characteristics across participants cause local models to diverge, leading to poor convergence and suboptimal global performance.

Methodology: Feature Norm Regularization

FNR-FL addresses data disparity by regularizing the feature space. The core idea is that the $L_2$ norm of features extracted by a neural network serves as an effective proxy for the underlying data distribution.

The FNR-FL process involves three key steps:

  1. Feature Norm Calculation: Participants compute the average $L_2$ norm of features for each class.
  2. Selective Regularization: The central server identifies "weak" participants—those with the lowest local test accuracy.
  3. Alignment Penalty: These participants add a regularization term to their local loss function, minimizing the distance between their local feature norms and the global average norms.

This approach forces local models to produce consistent feature distributions, effectively aligning update directions across the federation without requiring access to raw data.

Experimental Setup and Datasets

We evaluated FNR-FL against several state-of-the-art baselines, including FedAvg, MOON, FedProx, and SCAFFOLD. We utilized three computer vision benchmarks: CIFAR-10, MNIST, and FEMNIST. To simulate realistic decentralized environments, we introduced three types of data skews:

  • Label Distribution Skew: Imbalanced class proportions across participants.
  • Quantity Skew: Varying amounts of data per participant.
  • Feature Distribution Skew: Variations in image quality, simulated by adding Gaussian noise patches.
Figure 1: Visualization of label distribution skew (a) and quantity skew (b) across ten participants (P1-P10) for the CIFAR-10 dataset.
Figure 2: Examples of feature distribution skew, where noise patches are applied to images to simulate domain shifts.

Results and Discussion

FNR-FL demonstrates remarkable robustness across all skew types. On CIFAR-10 with feature distribution skew, FNR-FL achieved 99.76% accuracy, a massive improvement over FedAvg's 60.01%.

Beyond accuracy, FNR-FL improves training efficiency. Despite the additional regularization step, the algorithm reached convergence 11.40% faster than FedAvg. We also introduced two new efficiency metrics: $\kappa$ (accuracy per unit of time) and $\rho$ (accuracy per unit of data traffic), both of which FNR-FL dominated, showing it provides the best performance relative to computational and network costs.

Conclusion

FNR-FL provides a simple yet powerful solution to the non-i.i.d. challenge in Federated Learning. By leveraging feature norms as a proxy for distribution alignment, it achieves superior accuracy and faster convergence than existing methods. Future work will explore the integration of model compression and formal privacy frameworks like Differential Privacy to further enhance its real-world applicability.

Summary updated

BibTeX
@inproceedings{hu2024feature,
  author = {Ke Hu and Liyao Xiang and Peng Tang and Weidong Qiu},
  title = {{Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure