TL;DR
UPRTH (Unified Pretraining for Recommendation via Task Hypergraphs) is a new framework that moves away from text-based pretraining for recommender systems, opting instead to preserve complex data structures using hypergraphs. By unifying diverse auxiliary tasks—like predicting item categories or user relations—into a single hypergraph learning objective, it achieves massive performance gains (up to 139% on some metrics) and significantly improves recommendations for "cold-start" users.
What Problem Does This Paper Address?
Pretraining has revolutionized NLP and Computer Vision, but it’s tricky for graph-based recommender systems. Most recommenders rely on unique IDs (User 123, Item 456). These IDs are "meaningless" across different datasets, making it hard to transfer knowledge from one platform to another.
Furthermore, current multitask pretraining often relies on Large Language Models (LLMs) to "translate" user behavior into text. While clever, this introduces "linguistic bias"—the model might prioritize grammatically correct text over actual user preference—and destroys the rich structural information inherent in user-item interaction graphs. Finally, if you pretrain and finetune on the same dataset, the model often just "memorizes" the data (overfitting) rather than learning generalizable patterns.
Methods
The authors introduce UPRTH, which stands out through two main innovations:
- Task Hypergraphs: Instead of standard graphs (where an edge connects exactly two nodes), they use hypergraphs where an edge can connect any number of nodes. They generalize three types of tasks into this format:
- Recommendation: Connecting users who bought the same items.
- Relation Prediction: Connecting items that are substitutes for one another.
-
Attribute Prediction: Connecting all items that share a specific category or attribute. This allows the model to treat every "pretext" task as a simple hyperedge prediction problem, creating a unified learning pattern.
-
Transitional Attention (TA) Layer: Not all auxiliary tasks are equally useful for the final recommendation. The TA layer acts as a gatekeeper. It uses an attention mechanism to adaptively determine how much "knowledge" to transfer from an auxiliary task (like category prediction) to the main recommendation task. This prevents "over-generalization" and ensures the model focuses on the most relevant contextual cues.
Datasets
The framework was evaluated on three real-world datasets:
- XMarket-CN & XMarket-MX: Large-scale Amazon datasets covering China and Mexico, including item categories, ratings, and "bought together" relations.
- Steam: Data from the popular online game store, including user groups, game publishers, and friend networks.
Key Insights and Ideas
- Structure over Text: By keeping the data in a graph format rather than converting it to text, the model avoids the noise and bias of language models while capturing high-order relationships (e.g., a group of users all belonging to the same niche community).
- The Power of Unification: The "aha moment" here is that diverse tasks—which usually require different architectures—can all be viewed as "does this node belong in this hyperedge?" This simplicity allows for a much more robust pretraining phase.
- Selective Knowledge Transfer: The Transitional Attention layer recognizes that knowing a user's "friend group" might be more important for a game recommendation than knowing the "average rating" of a game. This discriminative learning is what prevents the model from being distracted by irrelevant data.
Key Results
- Superiority over Baselines: UPRTH consistently outperformed state-of-the-art models like LightGCN and other pretraining methods (GCC, SGL-PreRec). On the XMarket-CN dataset, it showed a staggering 139.11% improvement in NDCG@10 compared to the best baseline.
- Robustness in Cold-Start: One of the biggest wins was in "cold-start" scenarios (users with no history). By leveraging auxiliary tasks (like user attributes or social groups), UPRTH could make accurate guesses about what a new user would like, significantly outperforming models that only look at interaction history.
- Efficiency: Despite using complex hypergraphs, the authors show the model complexity remains $O(|V|^2)$, making it as efficient as standard matrix factorization.
Limitations and Future Work
The authors acknowledge that while they have unified several common tasks, the framework could be extended to even more diverse graph-based downstream tasks beyond just recommendation.
Additionally, while the paper focuses on ID-dependent datasets, the current version of UPRTH still faces challenges when transferring knowledge between datasets with completely disjoint ID spaces if no common attributes exist. Future work may involve exploring how to bridge these gaps using more "ID-agnostic" features within the hypergraph structure.
Summary updated