Generative Retrieval with Semantic Tree-Structured Item Identifiers via Contrastive Learning

Zihua Si, Zhongxiang Sun, Jiale Chen, Guozhang Chen, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, Jun Xu

Abstract

The retrieval phase is a vital component in recommendation systems, requiring the model to be effective and efficient. Recently, generative retrieval has become an emerging paradigm for document retrieval, showing notable performance. These methods enjoy merits like being end-to-end differentiable, suggesting their viability in recommendation. However, these methods fall short in efficiency and effectiveness for large-scale recommendations. To obtain efficiency and effectiveness, this paper introduces a generative retrieval framework, namely SEATER, which learns SEmAntic Tree-structured item identifiERs via contrastive learning. Specifically, we employ an encoder-decoder model to extract user interests from historical behaviors and retrieve candidates via tree-structured item identifiers. SEATER devises a balanced k-ary tree structure of item identifiers, allocating semantic space to each token individually. This strategy maintains semantic consistency within the same level, while distinct levels correlate to varying semantic granularities. This structure also maintains consistent and fast inference speed for all items. Considering the tree structure, SEATER learns identifier tokens' semantics, hierarchical relationships, and inter-token dependencies. To achieve this, we incorporate two contrastive learning tasks with the generation task to optimize both the model and identifiers. The infoNCE loss aligns the token embeddings based on their hierarchical positions. The triplet loss ranks similar identifiers in desired orders. In this way, SEATER achieves both efficiency and effectiveness. Extensive experiments on three public datasets and an industrial dataset have demonstrated that SEATER outperforms state-of-the-art models significantly.

Introduction

In modern recommendation systems, the retrieval phase is a high-stakes first step: it must sift through millions of items to find a few hundred candidates in milliseconds. Traditionally, this is achieved using dual-encoders and Approximate Nearest Neighbor (ANN) search. Recently, Generative Retrieval (e.g., DSI or TIGER) has emerged as a promising alternative where a model is trained to directly "generate" the identifier (ID) of the next item.

However, existing generative methods often suffer from three major flaws:

  1. Semantic Blindness: They treat item IDs as arbitrary strings, ignoring the inherent hierarchical relationships (e.g., Category $\to$ Sub-category $\to$ Item).
  2. Structural Neglect: The tree structure used to create IDs is often ignored during the actual neural network training.
  3. Efficiency Bottlenecks: Imbalanced trees lead to inconsistent path lengths, creating unpredictable latency in production environments.

The SEATER Framework

To address these challenges, we introduce SEATER (SEmAntic Tree-structured item identifiERs), a framework that treats ID generation as a traversal through a structured semantic space.

Balanced k-ary Tree Construction

Instead of arbitrary identifiers, SEATER clusters items using constrained $k$-means based on pre-trained collaborative filtering embeddings (e.g., from SASRec). This creates a balanced tree where every item ID has the exact same length. This ensures semantic consistency: tokens at the same level of the ID represent the same level of granularity, and search depth remains logarithmic ($O(\log_k N)$), guaranteeing fast and stable inference.

Multi-Task Contrastive Learning

SEATER's core innovation lies in its training objectives. Beyond the standard Generation Loss, we incorporate two novel contrastive tasks to align the model with the tree structure:

  • Alignment Loss ($\mathcal{L}_{ali}$): Using an infoNCE objective, this task ensures a "child" token's embedding is pulled closer to its "parent" token's embedding. This forces the model to internalize the hierarchical semantics of the identifiers.
  • Ranking Loss ($\mathcal{L}_{rank}$): Using a triplet loss, this task teaches the model to distinguish between similar items. It rewards the model for ranking identifiers that share long prefixes with the ground truth higher than those that do not.
Overview of SEATER's tree-structured identifiers and multi-task learning objectives, including generation, alignment, and ranking losses.

Experimental Evaluation

We evaluated SEATER on four diverse datasets: Yelp (dense, small-scale), Books (large-scale, sparse), News (MIND-small), and a massive industrial Micro-Video dataset (0.75M users, 85M interactions).

Performance and Efficiency

SEATER consistently outperformed state-of-the-art baselines, including dual-encoders like SASREC and generative models like TIGER. On the industrial Micro-Video dataset, SEATER achieved a 25.5% relative improvement in Recall@50 over the best baseline. Notably, SEATER achieves these results with a lightweight 1-layer Transformer architecture, making it highly viable for real-world deployment.

Performance comparison of SEATER against state-of-the-art dual-encoder, tree-based, and generative retrieval models across four datasets.

Ablation Studies

Ablation experiments confirmed the necessity of the proposed losses. Removing either the Alignment Loss or the Ranking Loss led to a significant performance drop, proving that explicitly training the model to understand the identifier structure is vital for effective generative retrieval.

Conclusion

SEATER demonstrates that item identifiers in generative retrieval should be treated as structured indices rather than simple text strings. By combining balanced tree construction with hierarchical contrastive learning, SEATER achieves a superior balance of retrieval accuracy and inference efficiency. Future work will explore handling "cold start" scenarios for new items and optimizing the branch number ($k$) for even larger item pools.

Summary updated

Abstract

The retrieval phase is a vital component in recommendation systems, requiring the model to be effective and efficient. Recently, generative retrieval has become an emerging paradigm for document retrieval, showing notable performance. These methods enjoy merits like being end-to-end differentiable, suggesting their viability in recommendation. However, these methods fall short in efficiency and effectiveness for large-scale recommendations. To obtain efficiency and effectiveness, this paper introduces a generative retrieval framework, namely SEATER, which learns SEmAntic Tree-structured item identifiERs via contrastive learning. Specifically, we employ an encoder-decoder model to extract user interests from historical behaviors and retrieve candidates via tree-structured item identifiers. SEATER devises a balanced k-ary tree structure of item identifiers, allocating semantic space to each token individually. This strategy maintains semantic consistency within the same level, while distinct levels correlate to varying semantic granularities. This structure also maintains consistent and fast inference speed for all items. Considering the tree structure, SEATER learns identifier tokens' semantics, hierarchical relationships, and inter-token dependencies. To achieve this, we incorporate two contrastive learning tasks with the generation task to optimize both the model and identifiers. The infoNCE loss aligns the token embeddings based on their hierarchical positions. The triplet loss ranks similar identifiers in desired orders. In this way, SEATER achieves both efficiency and effectiveness. Extensive experiments on three public datasets and an industrial dataset have demonstrated that SEATER outperforms state-of-the-art models significantly.
BibTeX
@inproceedings{si2024generative,
  author = {Zihua Si and Zhongxiang Sun and Jiale Chen and Guozhang Chen and Xiaoxue Zang and Kai Zheng and Yang Song and Xiao Zhang and Jun Xu},
  title = {{Generative Retrieval with Semantic Tree-Structured Item Identifiers via Contrastive Learning}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2024},
  doi = {10.1145/3673791.3698408},
}

Loading related content...

Figure