Abstract
Knowledge Graph Pre-training (KGP) has emerged as a powerful paradigm for enhancing various AI tasks with structured semantic information. However, existing KGP methods often struggle with transferability across domains and the high computational cost of full-model fine-tuning. This paper introduces Mudok, a novel framework designed for large-scale, multi-domain KG pre-training and efficient downstream adaptation. Mudok employs Collaborative Pre-training (CoPT) to learn robust item representations by mapping KG structures onto a universal textual space. For downstream tasks, it utilizes Prefix Prompt Tuning (PPT), which freezes the pre-trained model and only tunes a small set of prefix parameters. Evaluated on the new KPI benchmark, Mudok demonstrates significant performance improvements in recommendation and text understanding tasks while maintaining extreme parameter efficiency.
Introduction
The "pre-train then apply" paradigm in Knowledge Graphs (KGs) aims to capture rich semantics to boost downstream applications. Despite its potential, the field faces a "reproducibility and transferability" crisis. Industrial models often rely on proprietary data and learn shallow embeddings that are difficult to transfer to new domains. Conversely, academic research frequently utilizes small, idealized datasets that fail to capture real-world complexity. Furthermore, the standard practice of full-model fine-tuning is computationally expensive for large-scale applications. Mudok addresses these challenges by providing an open-source, efficient, and transferable framework for multi-domain KGP.
Methodology
The Mudok framework is divided into two primary stages: Collaborative Pre-training (CoPT) and Prefix Prompt Tuning (PPT).
Collaborative Pre-training (CoPT)
CoPT adopts an item-centric design, treating items as core entities. Input sequences are constructed from an item's attributes and values, which are then processed by a Transformer encoder. The model is optimized using a dual-objective approach: a Contrastive Loss (similar to SimCSE) to ensure representation robustness and a Knowledge Triple Loss to preserve structural KG constraints. A key innovation is freezing the initial textual features (extracted via BERT), which allows the model to use text as a universal "conduit" for cross-domain knowledge transfer.
Prefix Prompt Tuning (PPT)
To adapt to downstream tasks efficiently, Mudok employs PPT. Instead of fine-tuning the entire pre-trained KG model, the model parameters are frozen. A lightweight prefix prompt token is introduced for each item, and only these parameters are trained for specific tasks like recommendation or text classification. This "plug-and-play" approach enables seamless integration with various backbones, such as LightGCN for recommendation or RoBERTa for NLP tasks.
Experimental Setup
The authors introduced the KPI (Knowledge graph Pre-training as Infrastructure) benchmark to evaluate Mudok. This large-scale open-source benchmark includes datasets from Amazon (Digital Music, Movies & TV, Video Games) and Douban (Books, Music, Movies). The scale is significant: the Amazon KG contains over 540,000 entities, while the Douban KG features 3.6 million triples. The benchmark covers diverse tasks, including Recommendation (Rec), Question Answering (QA), and Review Prediction (RP).
Results and Discussion
Mudok consistently outperforms baseline methods across all evaluated tasks and backbones.
As shown in the results table, Mudok achieves substantial gains in recommendation metrics. For instance, using the LightGCN backbone on the Amazon-Music dataset, Mudok improves Recall@5 by 23.2%. The framework also shows versatility in NLP tasks, providing significant improvements over vanilla BERT, RoBERTa, and GPT-2 models by incorporating KG-aware representations that help language models "understand" specific product attributes.
Key insights from the evaluation include:
- Text as a Universal Bridge: Freezing textual embeddings allows the model to map KG structures onto a universal space, enabling strong performance even on Out-of-Domain (OOD) data.
- GNN Amplification: GNN-based recommendation models (like LightGCN) benefit more from prompt tuning because the prompt signal is amplified as it propagates through the user-item interaction graph.
- Extreme Efficiency: By only tuning prefix prompts, the number of trainable parameters is reduced to less than 1% of the total model size (as low as 0.33% for Amazon). This allows for training on million-scale KGs in under a minute on a single GPU.
Conclusion
Mudok successfully bridges the gap between massive multi-domain KGs and diverse downstream tasks. By leveraging text as a universal bridge and employing efficient prompt tuning, it offers a scalable and transferable solution for KGP. While the prefix tokens introduce a slight increase in training latency, the overall efficiency and performance gains are substantial. Future work will explore extending the framework to sequential and cross-domain recommendation scenarios and further unifying interfaces for broader application support.
Summary updated