Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning

Xiangrong Zhu, Guangyao Li, Wei Hu

Introduction

Knowledge Graphs (KGs) serve as the backbone for modern artificial intelligence, structuring information into relational triplets such as (Damian Lillard, plays for, Portland Trail Blazers). However, in practice, these graphs are often siloed across different organizations—such as IMDb, DBpedia, or private corporate databases—due to strict privacy regulations and the proprietary nature of data. Federated Learning (FL) has emerged as a promising solution, allowing these organizations to collaboratively train a shared model without exchanging raw data.

Despite its potential, FL for Knowledge Graphs faces two critical hurdles: data heterogeneity and the "right to be forgotten." Existing FL methods struggle when client data distributions differ significantly (e.g., one client specializing in sports and another in music), leading to model "drift." Furthermore, as data changes or privacy laws like GDPR are enforced, models must be able to "unlearn" specific triplets without the prohibitive cost of retraining from scratch. This paper introduces FedLU, a framework designed to address these challenges through mutual knowledge distillation and neuroscience-inspired unlearning.

Methodology

1. Learning via Mutual Knowledge Distillation (MKD)

Traditional Federated Learning typically relies on weight averaging (e.g., FedAvg). In highly heterogeneous environments, forcing local models to converge toward a single set of global weights often degrades performance because local optimizations conflict.

FedLU replaces weight averaging with Mutual Knowledge Distillation (MKD). In this setup, local and global embeddings are maintained in parallel and "communicate" by aligning their output distributions rather than their raw weights:

  • Global-to-Local: The global model acts as a teacher, providing a generalized perspective that helps local models improve their ability to predict unseen relations.
  • Local-to-Global: Local models act as teachers for the global model, allowing it to absorb specialized, domain-specific knowledge from the clients.

This approach allows for a "smooth knowledge exchange" that respects the nuances of local data while building a robust, unified global view.

2. Unlearning via Cognitive Neuroscience Principles

To erase specific triplets from a trained model, FedLU mimics the way the human brain forgets information, employing a two-step process:

  • Retroactive Interference: This step intentionally "confuses" the model regarding the data to be forgotten. It utilizes hard confusion (treating the forgotten triplet as a negative sample) and soft confusion (forcing the model to assign similar scores to the forgotten triplet and its negative samples). Soft confusion is particularly vital as it prevents the model from over-correcting and damaging the surrounding embedding space.
  • Passive Decay: Following interference, the model undergoes a round of mutual distillation using only the remaining (retaining) data. This allows the "memory traces" of the forgotten information to fade naturally while the model recovers any general performance lost during the interference phase.

Experimental Setup

The authors identified a gap in existing FL benchmarks, which often use random partitioning that fails to reflect real-world data silos. To address this, they introduced three new heterogeneous datasets based on FB15k-237:

  • FB15k-237-C3, C5, and C10: These datasets were generated using a clustering-based algorithm to ensure clients have distinct, non-overlapping relation sets, mimicking the specialized nature of real-world databases.
  • The framework was evaluated against standard random-partitioned versions (R3, R5, R10) and state-of-the-art federated methods like FedE and FedProx.

Key Results and Insights

FedLU consistently outperformed existing federated methods across all heterogeneous datasets. For instance, on the C3 dataset using the TransE model, FedLU achieved a Hits@10 of 57.53%, a significant improvement over the independent training baseline of 53.91%. The distillation approach also resulted in faster convergence compared to methods relying on direct weight aggregation.

Surgical Unlearning

The unlearning module proved highly effective at erasing specific facts while preserving the model's overall utility. On the ComplEx model, the Hits@1 score for the forgotten set plummeted from 59.16% to just 6.26% after the unlearning process. Crucially, the performance on the remaining test set remained stable, only dipping slightly from 25.38% to 23.49%, demonstrating that FedLU can "forget" without catastrophic loss of general knowledge.

Conclusion

FedLU represents a significant step forward in making Federated Knowledge Graph Embedding more practical and compliant with privacy standards. By moving from weight averaging to knowledge distillation, it effectively manages data heterogeneity. By looking to neuroscience for unlearning strategies, it provides a computationally efficient way to handle dynamic data and privacy requests. Future work will likely explore continual learning and unlearning, aiming to manage a constant stream of evolving data throughout the entire lifecycle of a Knowledge Graph.

Summary updated

Abstract

Federated Learning (FL) recently emerges as a paradigm to train a global machine learning model across distributed clients without sharing raw data. Knowledge Graph (KG) embedding represents KGs in a continuous vector space, serving as the backbone of many knowledge-driven applications. As a promising combination, federated KG embedding can fully take advantage of knowledge learned from different clients while preserving the privacy of local data. However, realistic problems such as data heterogeneity and knowledge forgetting still remain to be concerned. In this paper, we propose FedLU, a novel FL framework for heterogeneous KG embedding learning and unlearning. To cope with the drift between local optimization and global convergence caused by data heterogeneity, we propose mutual knowledge distillation to transfer local knowledge to global, and absorb global knowledge back. Moreover, we present an unlearning method based on cognitive neuroscience, which combines retroactive interference and passive decay to erase specific knowledge from local clients and propagate to the global model by reusing knowledge distillation. We construct new datasets for assessing realistic performance of the state-of-the-arts. Extensive experiments show that FedLU achieves superior results in both link prediction and knowledge forgetting.
BibTeX
@inproceedings{zhu2023heterogeneous,
  author = {Xiangrong Zhu and Guangyao Li and Wei Hu},
  title = {{Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning}},
  booktitle = {The Web Conference},
  year = {2023},
}
Figure