LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval

Kai Zhang, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, Daxin Jiang

TL;DR

LED (Lexicon-Enlightened Dense) is a training framework that injects keyword-matching capabilities into dense retrievers by distilling knowledge from lexicon-aware models. By using "weak" supervision—focusing on relative rankings rather than exact scores—LED bridges the gap between semantic understanding and precise term matching, outperforming both its teacher and standard dense models on major benchmarks.

What Problem Does This Paper Address?

Modern search systems often rely on Dense Retrievers (DRs), which represent queries and documents as condensed mathematical vectors. While DRs are excellent at capturing the "gist" or semantic meaning of a sentence, they are notoriously bad at "exact match" scenarios. They might overlook a specific entity (like "Thoros of Myr") or a technical term (like "active margin"), leading to irrelevant results.

Traditionally, engineers solve this by building Hybrid Retrievers that combine a DR with a keyword-based system like BM25. However, this is inefficient: it requires maintaining two separate indexes, running two searches, and figuring out how to merge the scores. The authors of this paper ask: Can we teach a single dense retriever to be "lexicon-aware" without losing its semantic strengths?

Methods

The authors propose the LED framework, which uses a lexicon-aware model (like SPLADE) as a "teacher" to guide a dense retriever "student." The training involves two novel strategies:

  1. Lexicon-Augmented Contrastive Learning: Standard dense models are trained to distinguish between a positive passage and "hard negatives" (passages that look relevant but aren't). LED uses the lexical teacher to find negatives that are lexically similar to the query but semantically wrong. These act as adversarial examples that force the dense model to pay closer attention to specific keywords.
  2. Pair-wise Rank-Consistent Regularization: Instead of forcing the student to match the teacher's exact scores (which can be too restrictive), LED uses a "weak" supervision signal. It simply requires that if the teacher ranks Passage A higher than Passage B, the student should try to do the same. This allows the student to learn lexical priorities while maintaining its own semantic properties.
Figure 1: The training framework of our LED retriever. The Lexical teacher is independently trained following a two-stage pro- cess. After warming up, LED is trained with negatives mined by self and two lexicon-aware retrievers for lexicon-augmented contrastive learning, during which the Lexical Model (𝜃lex2) enhances LED with pair-wise rank-consistent regularization.

Datasets

The model was evaluated on three widely recognized large-scale passage retrieval benchmarks:

  • MS MARCO: A massive dataset based on real Bing search queries.
  • TREC Deep Learning 2019 & 2020: Specialized tracks that provide more fine-grained relevance labels for a subset of MS MARCO queries.

Key Insights and Ideas

  • Weak Supervision is Key: The authors found that "soft" teaching (matching relative ranks) is more effective than "strict" teaching (matching exact score margins). Strict alignment often leads to training collapse because the dense model's architecture is fundamentally different from the sparse lexical model's architecture.
  • Lexical Negatives as Adversaries: By training on negatives that a keyword-matching system would rank highly, the dense model learns to "disentangle" mere keyword overlap from true relevance.
  • Complementary Strengths: Lexical distillation is complementary to standard cross-encoder distillation (Ranker Distillation). You can use both to get a "best of both worlds" retriever.
  • Distribution Shift: Visual analysis shows that LED's score distributions successfully shift toward the lexical teacher's behavior while still retaining the "smoothness" of a dense model.
Distributions of model predictions, showing how LED (green) shifts toward the Lexical teacher (red) compared to the original Dense model (blue).

Key Results

LED demonstrates significant improvements over standard dense retrievers across all benchmarks:

  • MS MARCO: LED achieved an MRR@10 of 39.6, outperforming its lexical teacher (38.3) and the dense baseline (38.1). When combined with ranker distillation (w/ RT), it reached 40.2.
  • TREC DL: On the 2019 dataset, LED (w/ RT) achieved an NDCG@10 of 74.4, a substantial jump over the baseline dense model's 71.8.
  • Efficiency: Because LED is a single dense model, it maintains the high inference speed and low memory footprint of standard DRs, avoiding the overhead of hybrid systems.
Table 1: Experimental results on MS MARCO, TREC DL 2019 (DL’19), and TREC DL 2020 (DL’20) datasets (%). We mark the best results in bold and the second-best underlined. Numbers marked with ‘*’ mean that the improvement is statistically significant compared with the baseline (t-test with 𝑝-value < 0.05).

Limitations and Future Work

The authors note that LED is still sensitive to the quality of its teacher. If the lexicon-aware teacher makes a mistake (e.g., ranking the ground truth passage very low), the LED model's performance on that specific query can suffer.

Future work could involve exploring "intractable" queries where neither dense nor lexical models currently perform well, perhaps by incorporating even more diverse sources of knowledge during the "enlightenment" phase.

Summary updated

Abstract

Retrieval models based on dense representations in semantic space have become an indispensable branch for first-stage retrieval. These retrievers benefit from surging advances in representation learning towards compressive global sequence-level embeddings. However, they are prone to overlook local salient phrases and entity mentions in texts, which usually play pivot roles in first-stage retrieval. To mitigate this weakness, we propose to make a dense retriever align a well-performing lexicon-aware representation model. The alignment is achieved by weakened knowledge distillations to enlighten the retriever via two aspects -- 1) a lexicon-augmented contrastive objective to challenge the dense encoder and 2) a pair-wise rank-consistent regularization to make dense model's behavior incline to the other. We evaluate our model on three public benchmarks, which shows that with a comparable lexicon-aware retriever as the teacher, our proposed dense one can bring consistent and significant improvements, and even outdo its teacher. In addition, we found our improvement on the dense retriever is complementary to the standard ranker distillation, which can further lift state-of-the-art performance.
BibTeX
@inproceedings{zhang2022led,
  author = {Kai Zhang and Chongyang Tao and Tao Shen and Can Xu and Xiubo Geng and Binxing Jiao and Daxin Jiang},
  title = {{LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval}},
  booktitle = {The Web Conference},
  year = {2022},
}
Figure