Abstract
Although pretraining has garnered significant attention and popularity in recent years, its application in graph-based recommender systems is relatively limited. It is challenging to exploit prior knowledge by pretraining in widely used ID-dependent datasets. On one hand, user-item interaction history in one dataset can hardly be transferred to other datasets through pretraining, where IDs are different. On the other hand, pretraining and finetuning on the same dataset leads to a high risk of overfitting. In this paper, we propose a novel multitask pretraining framework named Unified Pretraining for Recommendation via Task Hypergraphs. For a unified learning pattern to handle diverse requirements and nuances of various pretext tasks, we design task hypergraphs to generalize pretext tasks to hyperedge prediction. A novel transitional attention layer is devised to discriminatively learn the relevance between each pretext task and recommendation. Experimental results on three benchmark datasets verify the superiority of UPRTH. Additional detailed investigations are conducted to demonstrate the effectiveness of the proposed framework.
1. Introduction
Pretraining has revolutionized natural language processing and computer vision, yet its application in graph-based recommender systems remains limited. Traditional graph-based recommenders often rely on unique, dataset-specific IDs, which hinders knowledge transfer across platforms. Furthermore, many current multitask pretraining approaches rely on Large Language Models (LLMs) to translate user behaviors into text, potentially introducing linguistic bias and losing rich structural information inherent in user-item interaction graphs. To address these challenges, we propose UPRTH (Unified Pretraining for Recommendation via Task Hypergraphs), a framework that preserves complex data structures and avoids the pitfalls of text-based translation.
2. Methodology
The UPRTH framework introduces two primary innovations: Task Hypergraphs and the Transitional Attention (TA) Layer.
2.1 Task Hypergraphs
Standard graphs connect exactly two nodes per edge. In contrast, hypergraphs allow an edge to connect any number of nodes, capturing higher-order relationships. UPRTH generalizes three distinct pretext tasks into a unified hyperedge prediction problem:
- Recommendation: Connecting users who interact with the same items.
- Relation Prediction: Connecting items that serve as substitutes.
- Attribute Prediction: Connecting items that share specific categories or attributes.
2.2 Transitional Attention (TA) Layer
Not all auxiliary tasks contribute equally to the final recommendation objective. The TA layer employs an attention mechanism to adaptively weight the "knowledge" transferred from various pretext tasks. This discriminative learning ensures the model prioritizes the most relevant contextual cues, such as social group affiliations in gaming contexts, while mitigating the risk of overfitting to less relevant data.
3. Experimental Evaluation
We evaluated UPRTH on three real-world datasets: XMarket-CN and XMarket-MX (Amazon datasets with rich item relations and categories) and Steam (online game store data including social networks and publisher information).
4. Results and Discussion
UPRTH consistently outperforms state-of-the-art baselines, including LightGCN, GCC, and SGL-PreRec. Notably, on the XMarket-CN dataset, UPRTH achieved a 139.11% improvement in NDCG@10 over the best baseline.
4.1 Robustness in Cold-Start Scenarios
A significant advantage of UPRTH is its performance for "cold-start" users—those with little to no interaction history. By leveraging auxiliary information through the unified hypergraph structure, UPRTH can make accurate recommendations where history-only models fail.
4.2 Efficiency and Structure
By maintaining data in a graph format, UPRTH avoids the noise associated with LLM-based translation. Despite the complexity of hypergraphs, the model's complexity remains $O(|V|^2)$, making it as computationally efficient as standard matrix factorization techniques.
5. Conclusion
UPRTH demonstrates that unifying diverse pretext tasks through hypergraphs and selective attention significantly enhances recommendation performance and robustness. Future work will explore extending this framework to more diverse downstream tasks and improving cross-dataset transfer in disjoint ID spaces.
Summary updated