Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks

Yichi Zhang, Binbin Hu, Zhuo Chen, Lingbing Guo, Ziqi Liu, Zhiqiang Zhang, Lei Liang, Huajun Chen, Wen ZhangarXiv 2024 · DOI · arXiv

Abstract

Knowledge Graph Pre-training (KGP) has emerged as a powerful paradigm for enhancing various AI tasks with structured semantic information. However, existing KGP methods often struggle with transferability across domains and the high computational cost of full-model fine-tuning. This paper introduces Mudok, a novel framework designed for large-scale, multi-domain KG pre-training and efficient downstream adaptation. Mudok employs Collaborative Pre-training (CoPT) to learn robust item representations by mapping KG structures onto a universal textual space. For downstream tasks, it utilizes Prefix Prompt Tuning (PPT), which freezes the pre-trained model and only tunes a small set of prefix parameters. Evaluated on the new KPI benchmark, Mudok demonstrates significant performance improvements in recommendation and text understanding tasks while maintaining extreme parameter efficiency.

Introduction

The "pre-train then apply" paradigm in Knowledge Graphs (KGs) aims to capture rich semantics to boost downstream applications. Despite its potential, the field faces a "reproducibility and transferability" crisis. Industrial models often rely on proprietary data and learn shallow embeddings that are difficult to transfer to new domains. Conversely, academic research frequently utilizes small, idealized datasets that fail to capture real-world complexity. Furthermore, the standard practice of full-model fine-tuning is computationally expensive for large-scale applications. Mudok addresses these challenges by providing an open-source, efficient, and transferable framework for multi-domain KGP.

Methodology

The Mudok framework is divided into two primary stages: Collaborative Pre-training (CoPT) and Prefix Prompt Tuning (PPT).

The Mudok framework: Collaborative Pre-training (CoPT) and Prefix Prompt Tuning (PPT).

Collaborative Pre-training (CoPT)

CoPT adopts an item-centric design, treating items as core entities. Input sequences are constructed from an item's attributes and values, which are then processed by a Transformer encoder. The model is optimized using a dual-objective approach: a Contrastive Loss (similar to SimCSE) to ensure representation robustness and a Knowledge Triple Loss to preserve structural KG constraints. A key innovation is freezing the initial textual features (extracted via BERT), which allows the model to use text as a universal "conduit" for cross-domain knowledge transfer.

Prefix Prompt Tuning (PPT)

To adapt to downstream tasks efficiently, Mudok employs PPT. Instead of fine-tuning the entire pre-trained KG model, the model parameters are frozen. A lightweight prefix prompt token is introduced for each item, and only these parameters are trained for specific tasks like recommendation or text classification. This "plug-and-play" approach enables seamless integration with various backbones, such as LightGCN for recommendation or RoBERTa for NLP tasks.

Experimental Setup

The authors introduced the KPI (Knowledge graph Pre-training as Infrastructure) benchmark to evaluate Mudok. This large-scale open-source benchmark includes datasets from Amazon (Digital Music, Movies & TV, Video Games) and Douban (Books, Music, Movies). The scale is significant: the Amazon KG contains over 540,000 entities, while the Douban KG features 3.6 million triples. The benchmark covers diverse tasks, including Recommendation (Rec), Question Answering (QA), and Review Prediction (RP).

Results and Discussion

Mudok consistently outperforms baseline methods across all evaluated tasks and backbones.

Performance comparison of Mudok against base and PKGM methods across various recommendation backbones.

As shown in the results table, Mudok achieves substantial gains in recommendation metrics. For instance, using the LightGCN backbone on the Amazon-Music dataset, Mudok improves Recall@5 by 23.2%. The framework also shows versatility in NLP tasks, providing significant improvements over vanilla BERT, RoBERTa, and GPT-2 models by incorporating KG-aware representations that help language models "understand" specific product attributes.

Key insights from the evaluation include:

  • Text as a Universal Bridge: Freezing textual embeddings allows the model to map KG structures onto a universal space, enabling strong performance even on Out-of-Domain (OOD) data.
  • GNN Amplification: GNN-based recommendation models (like LightGCN) benefit more from prompt tuning because the prompt signal is amplified as it propagates through the user-item interaction graph.
  • Extreme Efficiency: By only tuning prefix prompts, the number of trainable parameters is reduced to less than 1% of the total model size (as low as 0.33% for Amazon). This allows for training on million-scale KGs in under a minute on a single GPU.

Conclusion

Mudok successfully bridges the gap between massive multi-domain KGs and diverse downstream tasks. By leveraging text as a universal bridge and employing efficient prompt tuning, it offers a scalable and transferable solution for KGP. While the prefix tokens introduce a slight increase in training latency, the overall efficiency and performance gains are substantial. Future work will explore extending the framework to sequential and cross-domain recommendation scenarios and further unifying interfaces for broader application support.

Summary updated

Abstract

Knowledge graphs (KGs) provide reliable external knowledge for a wide variety of AI tasks in the form of structured triples. Knowledge graph pre-training (KGP) aims to pre-train neural networks on large-scale KGs and provide unified interfaces to enhance different downstream tasks, which is a key direction for KG management, maintenance, and applications. Existing works often focus on purely research questions in open domains, or they are not open source due to data security and privacy in real scenarios. Meanwhile, existing studies have not explored the training efficiency and transferability of KGP models in depth. To address these problems, We propose a framework MuDoK to achieve multi-domain collaborative pre-training and efficient prefix prompt tuning to serve diverse downstream tasks like recommendation and text understanding. Our design is a plug-and-play prompt learning approach that can be flexibly adapted to different downstream task backbones. In response to the lack of open-source benchmarks, we constructed a new multi-domain KGP benchmark called KPI with two large-scale KGs and six different sub-domain tasks to evaluate our method and open-sourced it for subsequent research. We evaluated our approach based on constructed KPI benchmarks using diverse backbone models in heterogeneous downstream tasks. The experimental results show that our framework brings significant performance gains, along with its generality, efficiency, and transferability.
BibTeX
@misc{zhang2024multi,
  author = {Yichi Zhang and Binbin Hu and Zhuo Chen and Lingbing Guo and Ziqi Liu and Zhiqiang Zhang and Lei Liang and Huajun Chen and Wen Zhang},
  title = {{Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks}},
  year = {2024},
  eprint = {2405.13085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.13085},
}
Figure