TL;DR
This paper introduces SEATER, a novel generative retrieval framework for recommender systems that learns semantic, tree-structured item identifiers. By incorporating contrastive learning and a balanced k-ary tree structure, SEATER significantly improves retrieval effectiveness and efficiency compared to existing methods.
What Problem Does This Paper Address?
Modern recommender systems typically use a two-stage retrieve-then-rank approach. The retrieval stage is crucial for efficiency, as it narrows down a vast item catalog to a manageable subset. While recent generative retrieval methods have shown promise by treating identifier generation as a sequence-to-sequence task, they often overlook the inherent semantic structure of these identifiers. This leads to issues like imbalanced identifier trees, inconsistent lengths, and suboptimal training that doesn't fully leverage the structural information. Consequently, these methods can suffer from increased inference times and reduced effectiveness. SEATER aims to address this by explicitly modeling and optimizing the structure of item identifiers during the retrieval process.
Methods
SEATER employs an encoder-decoder architecture, similar to standard Transformer models, to capture user interests and generate item identifiers. The core novelty lies in how it constructs and trains these identifiers:
- Semantic Tree-Structured Identifiers: SEATER constructs identifiers using a balanced k-ary tree. This ensures that all item identifiers have a consistent length and that tokens at the same level of the tree share similar semantic granularity. The tree is built using hierarchical clustering on item embeddings (e.g., from SASREC), allowing it to incorporate prior collaborative filtering knowledge.
- Contrastive Learning for Structure Optimization: To imbue the identifiers with semantic meaning and hierarchical relationships, SEATER introduces two contrastive learning tasks:
- Alignment Loss: This loss encourages parent tokens in the identifier tree to be semantically close to their child tokens, while pushing them away from unrelated tokens. It uses an infoNCE loss to align token embeddings based on their hierarchical positions.
- Ranking Loss: This loss trains the model to distinguish between similar identifiers by learning to rank them according to their shared prefix lengths. It uses a triplet loss to ensure that identifiers with more shared tokens are ranked higher.
- Multi-Task Training: The model is trained end-to-end using a combination of the generative (sequence-to-sequence) loss, the alignment loss, and the ranking loss.
Datasets
The paper evaluates SEATER on four datasets:
- Public Datasets:
- Yelp
- Books (Amazon review dataset subset)
- News (MIND-small subset)
- Industrial Dataset:
- Micro-Video (a newly collected dataset from a commercial app, noted for its high sparsity).
Key Insights and Ideas
- Identifier Structure Matters: The paper strongly argues that simply generating identifiers as text is insufficient. Explicitly modeling and optimizing their tree structure and semantic relationships is key to improving retrieval performance.
- Balanced Trees for Efficiency and Consistency: Using a balanced k-ary tree for identifiers leads to consistent semantic granularity across tokens at the same level and significantly reduces inference time by ensuring uniform identifier lengths.
- Contrastive Learning as a Structural Enhancer: The two contrastive learning tasks are crucial for teaching the model to understand the hierarchical nature and semantic nuances of the tree-structured identifiers, going beyond simple text generation.
- Leveraging Prior Knowledge: Incorporating item embeddings from pre-trained models (like SASREC) into the identifier construction process effectively injects collaborative filtering insights without additional training overhead for those embeddings.
- Single-Layer Efficiency: SEATER achieves state-of-the-art results with a single-layer encoder-decoder, demonstrating a significant efficiency advantage over multi-layer generative models like TIGER.
Key Results
- State-of-the-Art Performance: SEATER consistently outperforms various baseline models (dual-encoder, tree-based indexing, and generative) across all four datasets in terms of NDCG, HR, and Recall metrics.
- Significant Improvements: The paper reports substantial relative improvements, for example, in R@50 on the Micro-Video dataset (25.50%).
- Efficiency Gains: SEATER demonstrates superior inference speed compared to other tree-based and generative methods, as validated by complexity analysis and empirical results.
- Ablation Study Validation: Removing either the alignment or ranking loss, or using imbalanced identifiers (like DSI/NCI), leads to a significant performance drop, underscoring the importance of SEATER's proposed components.
Limitations and Future Work
- Deeper Models: While SEATER performs well with a single layer, the paper suggests that deeper models could potentially yield further improvements, especially on larger datasets, though this might come at a computational cost.
- Parameter Count Trade-offs: The paper notes that increasing the number of transformer layers leads to linear growth in computational complexity, and SEATER's single-layer design strikes a good balance between performance and efficiency.
- Hyperparameter Tuning: The performance of SEATER is sensitive to hyperparameter choices (e.g., loss coefficients, branch number k), and extensive tuning is required.
- Identifier Construction: While the balanced tree and SASREC embeddings are effective, exploring other methods for identifier construction and embedding sources could be a future direction.
Summary updated