Generative Retrieval with Semantic Tree-Structured Identifiers and Contrastive Learning

Zihua Si, Zhongxiang Sun, Jiale Chen, Guozhang Chen, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, Jun Xu, Kun Gai

TL;DR

This paper introduces SEATER, a novel generative retrieval framework for recommender systems that learns semantic, tree-structured item identifiers. By incorporating contrastive learning and a balanced k-ary tree structure, SEATER significantly improves retrieval effectiveness and efficiency compared to existing methods.

What Problem Does This Paper Address?

Modern recommender systems typically use a two-stage retrieve-then-rank approach. The retrieval stage is crucial for efficiency, as it narrows down a vast item catalog to a manageable subset. While recent generative retrieval methods have shown promise by treating identifier generation as a sequence-to-sequence task, they often overlook the inherent semantic structure of these identifiers. This leads to issues like imbalanced identifier trees, inconsistent lengths, and suboptimal training that doesn't fully leverage the structural information. Consequently, these methods can suffer from increased inference times and reduced effectiveness. SEATER aims to address this by explicitly modeling and optimizing the structure of item identifiers during the retrieval process.

Methods

SEATER employs an encoder-decoder architecture, similar to standard Transformer models, to capture user interests and generate item identifiers. The core novelty lies in how it constructs and trains these identifiers:

  • Semantic Tree-Structured Identifiers: SEATER constructs identifiers using a balanced k-ary tree. This ensures that all item identifiers have a consistent length and that tokens at the same level of the tree share similar semantic granularity. The tree is built using hierarchical clustering on item embeddings (e.g., from SASREC), allowing it to incorporate prior collaborative filtering knowledge.
  • Contrastive Learning for Structure Optimization: To imbue the identifiers with semantic meaning and hierarchical relationships, SEATER introduces two contrastive learning tasks:
  • Alignment Loss: This loss encourages parent tokens in the identifier tree to be semantically close to their child tokens, while pushing them away from unrelated tokens. It uses an infoNCE loss to align token embeddings based on their hierarchical positions.
  • Ranking Loss: This loss trains the model to distinguish between similar identifiers by learning to rank them according to their shared prefix lengths. It uses a triplet loss to ensure that identifiers with more shared tokens are ranked higher.
  • Multi-Task Training: The model is trained end-to-end using a combination of the generative (sequence-to-sequence) loss, the alignment loss, and the ranking loss.

Datasets

The paper evaluates SEATER on four datasets:

  • Public Datasets:
    • Yelp
    • Books (Amazon review dataset subset)
    • News (MIND-small subset)
  • Industrial Dataset:
    • Micro-Video (a newly collected dataset from a commercial app, noted for its high sparsity).

Key Insights and Ideas

  • Identifier Structure Matters: The paper strongly argues that simply generating identifiers as text is insufficient. Explicitly modeling and optimizing their tree structure and semantic relationships is key to improving retrieval performance.
  • Balanced Trees for Efficiency and Consistency: Using a balanced k-ary tree for identifiers leads to consistent semantic granularity across tokens at the same level and significantly reduces inference time by ensuring uniform identifier lengths.
  • Contrastive Learning as a Structural Enhancer: The two contrastive learning tasks are crucial for teaching the model to understand the hierarchical nature and semantic nuances of the tree-structured identifiers, going beyond simple text generation.
  • Leveraging Prior Knowledge: Incorporating item embeddings from pre-trained models (like SASREC) into the identifier construction process effectively injects collaborative filtering insights without additional training overhead for those embeddings.
  • Single-Layer Efficiency: SEATER achieves state-of-the-art results with a single-layer encoder-decoder, demonstrating a significant efficiency advantage over multi-layer generative models like TIGER.

Key Results

  • State-of-the-Art Performance: SEATER consistently outperforms various baseline models (dual-encoder, tree-based indexing, and generative) across all four datasets in terms of NDCG, HR, and Recall metrics.
  • Significant Improvements: The paper reports substantial relative improvements, for example, in R@50 on the Micro-Video dataset (25.50%).
  • Efficiency Gains: SEATER demonstrates superior inference speed compared to other tree-based and generative methods, as validated by complexity analysis and empirical results.
  • Ablation Study Validation: Removing either the alignment or ranking loss, or using imbalanced identifiers (like DSI/NCI), leads to a significant performance drop, underscoring the importance of SEATER's proposed components.

Limitations and Future Work

  • Deeper Models: While SEATER performs well with a single layer, the paper suggests that deeper models could potentially yield further improvements, especially on larger datasets, though this might come at a computational cost.
  • Parameter Count Trade-offs: The paper notes that increasing the number of transformer layers leads to linear growth in computational complexity, and SEATER's single-layer design strikes a good balance between performance and efficiency.
  • Hyperparameter Tuning: The performance of SEATER is sensitive to hyperparameter choices (e.g., loss coefficients, branch number k), and extensive tuning is required.
  • Identifier Construction: While the balanced tree and SASREC embeddings are effective, exploring other methods for identifier construction and embedding sources could be a future direction.

Summary updated

Abstract

In recommender systems, the retrieval phase is at the first stage and of paramount importance, requiring both effectiveness and very high efficiency. Recently, generative retrieval methods such as DSI and NCI, offering the benefit of end-to-end differentiability, have become an emerging paradigm for document retrieval with notable performance improvement, suggesting their potential applicability in recommendation scenarios. A fundamental limitation of these methods is their approach of generating item identifiers as text inputs, which fails to capture the intrinsic semantics of item identifiers as indices. The structural aspects of identifiers are only considered in construction and ignored during training. In addition, generative retrieval methods often generate imbalanced tree structures and yield identifiers with inconsistent lengths, leading to increased item inference time and sub-optimal performance. We introduce a novel generative retrieval framework named SEATER, which learns SEmAntic Tree-structured item identifiERs using an encoder-decoder structure. To optimize the structure of item identifiers, SEATER incorporates two contrastive learning tasks to ensure the alignment of token embeddings and the ranking orders of similar identifiers. In addition, SEATER devises a balanced k-ary tree structure of item identifiers, thus ensuring consistent semantic granularity and inference efficiency. Extensive experiments on three public datasets and an industrial dataset have demonstrated that SEATER outperforms a number of state-of-the-art models significantly.
BibTeX
@inproceedings{si2024generative,
  author = {Zihua Si and Zhongxiang Sun and Jiale Chen and Guozhang Chen and Xiaoxue Zang and Kai Zheng and Yang Song and Xiao Zhang and Jun Xu and Kun Gai},
  title = {{Generative Retrieval with Semantic Tree-Structured Identifiers and Contrastive Learning}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2024},
  doi = {10.1145/3673791.3698408},
}

Loading related content...

Figure