Improving Federated Relational Data Modeling via Basis Alignment and Weight Penalty

Yilun Lin, Chaochao Chen, Cen Chen, Li WangarXiv 2020 · OpenReview

Introduction

Federated Learning (FL) has revolutionized decentralized model training in domains like computer vision and natural language processing. However, its application to relational data—specifically Knowledge Graphs (KGs)—remains a significant challenge. Relational data is inherently "non-separable"; partitioning a graph across clients often destroys critical structural information, leading to extreme data heterogeneity. This paper introduces FedAlign, a novel framework designed to mitigate these issues through distribution-based basis alignment and optimization smoothing.

Challenges in Federated Relational Modeling

The authors identify two primary hurdles that prevent standard FL algorithms from succeeding on graph data:

  1. Objective Divergence: Graphs cannot be easily batched without losing context. In a federated setting, each client’s local graph is typically a biased sample of the global topology. This leads to local models drifting significantly away from the global objective during training.
  2. Objective Unsmoothness: The loss surfaces of Graph Neural Networks (GNNs) are often jagged. Standard FL algorithms like FedAVG rely on Lipschitz smoothness for convergence, a property frequently absent in complex relational modeling.

The FedAlign Framework

To address these challenges, FedAlign builds upon a Federated Relational Graph Convolutional Network (Fed-RGCN) and introduces two core innovations:

Basis Alignment via Optimal Transport (OT)

RGCNs utilize basis decomposition to handle numerous relations efficiently, where weights are represented as linear combinations of shared basis matrices. FedAlign moves beyond simple averaging by employing the Sinkhorn distance—an efficient implementation of Optimal Transport. By treating model weights as distributions ("piles of mass"), the framework aligns local and global bases more robustly than Euclidean-based methods, effectively managing structural discrepancies between clients.

The FedAlign architecture, illustrating the interaction between the trusted server and participants using basis decomposition.

1-Lipschitz Weight Penalty

To stabilize the optimization process, the authors implement a gradient penalty inspired by Generative Adversarial Network (GAN) research. By adding a term to the loss function that encourages the gradient norm to remain near 1, the model enforces "quasi-Lipschitz" continuity. This smooths the loss landscape, preventing the erratic updates common in GNN training.

Experimental Evaluation

FedAlign was evaluated using three standard Resource Description Framework (RDF) datasets, synthetically partitioned into 10 clients:

  • AIFB: Academic entities and relations.
  • MUTAG: Complex molecular graphs.
  • BGS: Hierarchical geological rock types.

Despite random splitting, the resulting local graphs exhibited high heterogeneity in node and edge density, providing a rigorous test for federated robustness.

Results and Discussion

FedAlign consistently outperformed baselines such as FedAVG and FedProx across all metrics.

Accuracy vs. Epoch performance across AIFB, MUTAG, and BGS datasets, comparing FedAlign against standard federated learning baselines.
  • Performance Gains: On the AIFB dataset, FedAlign reached 60.56% accuracy, surpassing FedAVG (56.94%) and FedProx (57.50%).
  • Enhanced Stability: The inclusion of the weight penalty (denoted as FedAlign-L in results) led to significantly smoother convergence curves, particularly on the complex MUTAG and BGS datasets.
  • Personalization: The basis alignment mechanism allowed the global model to better adapt to the unique structural characteristics of individual participant graphs.

Conclusion

FedAlign represents a significant step forward in federated graph modeling by treating weights as distributions and enforcing optimization smoothness. While a "federation gap" remains due to the inherent information loss when splitting graphs, the use of Optimal Transport and Lipschitz penalties provides a robust foundation for future research into privacy-preserving entity alignment and global topology preservation.

Summary updated

Abstract

Federated learning (FL) has attracted increasing attention in recent years. As a privacy-preserving collaborative learning paradigm, it enables a broader range of applications, especially for computer vision and natural language processing tasks. However, to date, there is limited research of federated learning on relational data, namely Knowledge Graph (KG). In this work, we present a modified version of the graph neural network algorithm that performs federated modeling over KGs across different participants. Specifically, to tackle the inherent data heterogeneity issue and inefficiency in algorithm convergence, we propose a novel optimization algorithm, named FedAlign, with 1) optimal transportation (OT) for on-client personalization and 2) weight constraint to speed up the convergence. Extensive experiments have been conducted on several widely used datasets. Empirical results show that our proposed method outperforms the state-of-the-art FL methods, such as FedAVG and FedProx, with better convergence.
BibTeX
@misc{lin2020improving,
  author = {Yilun Lin and Chaochao Chen and Cen Chen and Li Wang},
  title = {{Improving Federated Relational Data Modeling via Basis Alignment and Weight Penalty}},
  year = {2020},
  eprint = {2011.11369},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.11369},
}
Figure