Towards Distributionally Balanced End-to-End Generative Retrieval

Qi Zhang (张奇), Yuxuan Liu, Tianchi Yang, Zihan Zhang, Minghui Song, Haizhen Huang, Weiwei Deng, Feng Sun, Qi ZhangWWW 2025 · DOI

TL;DR

This paper introduces ASI++, a novel end-to-end generative retrieval framework that tackles the challenge of imbalanced document ID utilization in existing methods. By introducing novel criteria for distributionally balanced ID assignments and enhancing dense representations, ASI++ significantly improves retrieval performance and achieves more efficient use of the indexing space.

What Problem Does This Paper Address?

Traditional information retrieval systems often rely on complex indexing mechanisms that can suffer from a "semantic gap" between pre-defined document identifiers (IDs) and the actual retrieval task. Generative retrieval, a newer paradigm, aims to bridge this gap by learning to generate document IDs directly. However, existing end-to-end generative retrieval methods struggle with the long-tailed distribution of real-world data, leading to inefficient and unbalanced utilization of the ID space. This means some IDs might point to too many documents, diluting relevance, while others remain unused. This paper addresses this critical issue by proposing ASI++, a method that simultaneously learns balanced ID assignments and improves retrieval accuracy within a unified end-to-end framework.

Methods

ASI++ builds upon the ASI framework by introducing several key innovations:

  • Distributionally Balanced Criterion: This novel criterion aims to spread learned IDs more evenly across the entire ID space, preventing over-concentration and promoting better utilization. It uses a "distribution density loss" to achieve this balance.
  • Representation Bottleneck Criterion: To help the semantic indexing module learn efficient and balanced IDs, ASI++ introduces a criterion that enhances the quality of the dense representations generated by the encoder. This makes it easier for subsequent modules to process and assign IDs.
  • Information Consistency Criterion: This criterion, grounded in information theory, jointly optimizes the processes of ID assignment and retrieval. It acts as a semantic compression mechanism, ensuring that only crucial information for retrieval performance is retained in the dense bottleneck representations.

The paper also explores different implementations for the semantic indexing module, including differentiable Product Quantization (PQ), Residual Quantization (RQ), and an MLP-based neural quantization.

Datasets

The paper evaluates ASI++ on two document retrieval datasets:

  • MS MARCO: A well-studied public dataset for document ranking.
  • ADS: An industrial dataset from Bing sponsored search, consisting of query-ad pairs.

Key Insights and Ideas

  • End-to-End Learning is Key: The paper reinforces the benefits of end-to-end training for generative retrieval, allowing for a more cohesive learning process between ID assignment and retrieval.
  • Distributional Balance Matters: Simply assigning IDs isn't enough; how those IDs are distributed across the entire space significantly impacts retrieval performance. ASI++'s focus on this balance is a core contribution.
  • Dense Representations as Bottlenecks: The paper highlights the importance of the quality of dense representations generated by the encoder. Enhancing these representations directly benefits the downstream ID assignment and retrieval tasks.
  • Information Theory for Cohesion: Applying information-theoretic principles helps to unify the optimization objectives for different components of the model, leading to a more robust and effective system.
  • Hierarchical Structure in Learned IDs: The learned indexing space naturally forms a hierarchical semantic structure, similar to human-defined taxonomies, which aids in interpretability.

Key Results

  • State-of-the-Art Performance: ASI++ significantly outperforms existing sparse, dense, and generative retrieval baselines on the MS MARCO dataset, achieving state-of-the-art (SOTA) performance in metrics like R@1 and MRR.
  • Improved ID Assignment: ASI++ demonstrates a more dispersed and balanced distribution of document IDs compared to its predecessor, ASI, leading to better retrieval quality.
  • Robustness on New Documents: The model shows strong performance even on "New Semantic" documents (those with unseen IDs), indicating good generalizability.
  • Efficiency: The end-to-end nature and efficient one-to-many mapping contribute to efficient retrieval, especially on large-scale corpora.
  • Ablation Studies Confirm Contributions: Removing any of the proposed criteria (information consistency, representation bottleneck, distribution balancing) leads to performance degradation, underscoring their individual importance.

Limitations and Future Work

  • ID Comprehensibility: While ASI++ learns a hierarchical semantic structure, individual IDs might still be difficult for humans to comprehend directly, unlike natural language identifiers.
  • Ranking within IDs: The current framework doesn't explicitly handle ranking documents that share the same ID. The authors suggest this could be a promising direction for future work.
  • Negative Mining: The training of ASI++ currently relies on in-batch negatives. Exploring more sophisticated negative mining strategies could further improve performance.
  • Integration with LLMs: The authors suggest that ASI++ could be integrated with Large Language Models (LLMs) for retrieval-augmented generation, potentially reducing hallucinations and improving knowledge retrieval.

Summary updated

BibTeX
@inproceedings{2025towards,
  author = {Qi Zhang (张奇) and Yuxuan Liu and Tianchi Yang and Zihan Zhang and Minghui Song and Haizhen Huang and Weiwei Deng and Feng Sun and Qi Zhang},
  title = {{Towards Distributionally Balanced End-to-End Generative Retrieval}},
  booktitle = {The Web Conference},
  year = {2025},
  doi = {10.1145/3701716.3715263},
}

Loading related content...

Figure