UnifieR: A Unified Retriever for Large-Scale Retrieval

Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Guodong Long, Kai Zhang, Daxin Jiang

TL;DR

UnifieR is a novel retrieval framework that bridges the gap between dense-vector and lexicon-based retrieval by training a single neural encoder with dual representing capabilities. By employing a "dual-consistency" learning strategy, it achieves state-of-the-art performance on large-scale benchmarks while maintaining the efficiency required for real-world production systems.

What Problem Does This Paper Address?

In the world of large-scale information retrieval, researchers typically choose between two paradigms. Dense-vector retrieval embeds queries and documents into a continuous semantic space, excelling at capturing broad meaning but often missing specific keywords. Conversely, lexicon-based neural retrieval (like SPLADE) focuses on word-level importance and exact matches but can struggle with broader context.

While hybrid methods exist, they usually just combine the final scores of two independent models. This "surface-level" interaction fails to exploit the deep, complementary relationship between global sequence-level compression and local word-level context. The authors of UnifieR argue that by unifying these views within a single model during training, each paradigm can act as a teacher for the other, leading to a more robust and accurate retriever.

Methods

The core of UnifieR is a shared neural encoder that produces two distinct types of embeddings simultaneously.

  1. Shared Contextualization: The model starts with a shared Transformer-based feature extractor to ensure consistent semantic sharing between the two views.
  2. Dual Representing Modules:
    • Dense Module: Produces a sequence-level dense vector (global view).
  3. Lexicon Module: Produces a sparse, high-dimensional vector for keyword matching (local view). Crucially, this module is "global-aware"—it uses the dense vector as a "soft-prompt" to guide word-level weighting.
Figure 2: The encoder in Unifier.

To train this dual-headed beast, the authors propose Dual-Consistency Learning, a two-stage process:

  • Warmup Stage: The model is trained using standard contrastive loss with BM25-mined negatives.
  • Continual Training Stage: This stage introduces two novel strategies:
  • Negative-bridged Self-Adversarial: Hard negatives discovered by the dense module are used to train the lexicon module, and vice versa. This cross-pollination forces both modules to resolve each other's weaknesses.
  • Agreement-based Self-Regularization: A KL-divergence loss encourages the probability distributions of both modules to align, ensuring they provide a consistent view of document relevance.

Finally, they introduce a Uni-retrieval scheme: a fast pipeline where the lexicon module fetches a candidate set via an inverted index, which is then re-scored by the dense module.

Datasets

The authors evaluated UnifieR on three major benchmarks:

  • MS-Marco Passage Retrieval: The industry standard for large-scale web search.
  • TREC Deep Learning 2019: A benchmark focused on high-precision ranking.
  • BEIR (Benchmarking IR): A collection of 12 diverse datasets (e.g., COVID-19 research, financial Q&A, legal documents) used to test how well the model generalizes to new domains without further training.

Key Insights and Ideas

  • Global Guides Local: One of the most interesting findings is that providing the lexicon module with the global dense vector significantly improves its performance. The dense vector acts as a conceptual anchor, helping the model decide which words are truly important for the overall topic.
  • Cross-Paradigm Distillation: The dual-consistency strategy effectively allows the model to perform "self-distillation." The lexicon module, which is naturally better at keyword overlap, helps the dense module learn finer-grained features that it might otherwise ignore.
  • Efficiency without Sacrifice: The uni-retrieval scheme proves that you don't need to run two full retrieval passes. By using the sparse index as a filter, the dense re-scoring adds negligible latency, making it practical for real-time use.

Key Results

UnifieR achieved impressive numbers across the board:

  • State-of-the-Art: On MS-Marco Dev, the Uni-retrieval scheme reached 40.7% MRR@10, outperforming existing hybrid and single-paradigm models.
  • Transferability: On the BEIR benchmark, UnifieR showed strong zero-shot performance, achieving an average nDCG@10 of 44.5, which is competitive with much larger models or those using expensive rerankers.
  • Consistency: Figure 4 demonstrates that the "Dual-trn" (dual training) approach consistently lifts the performance of both the dense and lexicon heads compared to training them independently.
Figure 4: Verifying consistency of dual rep- resenting modules. ‘trn’ denotes ‘training’.

Limitations and Future Work

The authors identify two main hurdles:

  1. Initialization Constraints: Because of its specific dual-module architecture, UnifieR currently requires initialization from a limited set of pre-trained models (like coCondenser) that share a similar structure.
  2. Infrastructure Overhead: While retrieval is fast, the system requires infrastructure to store and manage both dense and sparse indexes for the entire document collection, which can be resource-intensive for massive datasets.

Future work may explore better aggregation methods for the dual scores and expanding the framework to support a wider variety of pre-trained language model architectures.

Summary updated

Abstract

Large-scale retrieval is to recall relevant documents from a huge collection given a query. It relies on representation learning to embed documents and queries into a common semantic encoding space. According to the encoding space, recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. These two paradigms unveil the PLMs' representation capability in different granularities, i.e., global sequence-level compression and local word-level contexts, respectively. Inspired by their complementary global-local contextualization and distinct representing views, we propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability. Experiments on passage retrieval benchmarks verify its effectiveness in both paradigms. A uni-retrieval scheme is further presented with even better retrieval quality. We lastly evaluate the model on BEIR benchmark to verify its transferability.
BibTeX
@inproceedings{shen2022unifier,
  author = {Tao Shen and Xiubo Geng and Chongyang Tao and Can Xu and Guodong Long and Kai Zhang and Daxin Jiang},
  title = {{UnifieR: A Unified Retriever for Large-Scale Retrieval}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2022},
}
Figure