Subspace Constraint and Contribution Estimation for Heterogeneous Federated Learning

Xiangtao Zhang, Sheng Li, Ao Li, Yipeng Liu, Fan Zhang, Ce Zhu, Le ZhangCVPR 2025 · DOI

Abstract

Federated Learning (FL) in heterogeneous environments (HFL) is often compromised by model architecture diversity and non-IID data distributions. We present FedSCE (Subspace Constraint and Estimation), a novel framework designed to mitigate local overfitting and aggregation bias. FedSCE employs a hierarchical subspace constraint to regularize local updates and a dual-space contribution estimation mechanism to weight client updates effectively. Experimental results across multiple datasets demonstrate that FedSCE significantly outperforms existing HFL methods in both accuracy and convergence speed.

Introduction

Standard Federated Learning assumes uniform model architectures and independent, identically distributed (IID) data. In practice, HFL must contend with model heterogeneity, where clients possess varying hardware capabilities, and data heterogeneity, characterized by label and feature skew. While auxiliary models can facilitate knowledge transfer between heterogeneous clients, they are prone to overfitting on local, skewed datasets. Furthermore, conventional aggregation methods like simple averaging fail to account for the varying quality and novelty of client contributions.

Methodology

FedSCE introduces two core components to address these challenges:

Hierarchical Subspace Constraint (LSL)

To prevent local models from overfitting to noise in small, skewed datasets, FedSCE restricts parameter updates to a low-dimensional subspace. This subspace is dynamically updated using Singular Value Decomposition (SVD) based on the trajectory of previous training rounds. By limiting the degrees of freedom, the model is forced to learn more generalized features, enhancing its performance on unseen data.

Auxiliary Model Contribution Estimation (CE)

Instead of uniform weighting, FedSCE estimates the value of each client's contribution by measuring the "update distance" of a shared auxiliary model. This distance is calculated in both the parameter space and the feature representation space. Clients that induce a larger shift in the auxiliary model—indicating the presence of novel information—are assigned higher weights during the global aggregation phase.

The FedSCE framework architecture, detailing the local update with subspace constraints and the collaborative update with contribution estimation.

Experimental Evaluation

The effectiveness of FedSCE was evaluated on several benchmark datasets, including Digit5 and Office10 for feature skew, and CIFAR-10/100 for label skew. The experimental setup included a diverse set of architectures such as ResNet and Vision Transformers (ViT).

Key Results

  • Superior Accuracy: FedSCE consistently outperformed state-of-the-art baselines like FedKD and FedProto. Notably, it achieved a 3.87% accuracy gain on the Office10 dataset.
  • Robustness to Skew: The method maintained high performance even under extreme label skew ($Dir_{0.1}$), demonstrating its ability to handle highly non-IID data.
  • Enhanced Convergence: Accuracy curves indicate that FedSCE reaches higher performance levels faster than traditional averaging methods.
Comparison of auxiliary model accuracy and convergence curves, highlighting the advantages of contribution estimation (CE) over simple averaging (Avg).

Conclusion

FedSCE provides a robust solution for Heterogeneous Federated Learning by combining subspace regularization with informed aggregation. By focusing on meaningful structural updates and accurately estimating client contributions, the framework achieves state-of-the-art performance. Future research will focus on automating the selection of layers for subspace projection and exploring applications in natural language processing.

Summary updated

BibTeX
@inproceedings{zhang2025subspace,
  author = {Xiangtao Zhang and Sheng Li and Ao Li and Yipeng Liu and Fan Zhang and Ce Zhu and Le Zhang},
  title = {{Subspace Constraint and Contribution Estimation for Heterogeneous Federated Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01921},
}
Figure