In modern information retrieval, there is a fundamental tension between how well a model understands a query and how fast it can search a billion documents.
The most accurate models, known as Cross-Encoders, perform "joint encoding." They feed the query and the document into a Transformer simultaneously, allowing every word in the query to interact with every word in the document through self-attention. As Jimmy Lin, Andrew Yates, and Rodrigo Nogueira 0001 detail in Pretrained Transformers for Text Ranking: BERT and Beyond, this deep interaction is the "BERT revolution" that moved search from keyword matching to semantic understanding. However, Cross-Encoders are computationally ruinous; you cannot run a 12-layer Transformer for every document in a massive index at query time.
To solve this, we use Bi-Encoders (or Dual-Towers). These models encode the query and document independently into single vectors. Search then becomes a simple dot product, which is lightning-fast. The catch? Because the query and document never "see" each other during encoding, the model loses the nuanced, token-level interactions that make Cross-Encoders so effective.
Knowledge Distillation (KD) is the bridge across this gap. It allows us to keep the speed of the Bi-Encoder while "teaching" it the wisdom of the Cross-Encoder.
The Intuition: Soft Labels are Better Teachers
In standard supervised learning, we train models on "hard labels" (e.g., this document is a 1, that one is a 0). But relevance is rarely binary. A document might be "almost relevant" or "relevant for a different reason."
When a Cross-Encoder (the teacher) scores a document, it produces a continuous "soft label" (e.g., 0.87). This score contains far more information than a binary 1. It tells the student model not just that a document is relevant, but how relevant it is compared to others. Jian Jiao and colleagues demonstrated in TwinBERT that by training a Bi-Encoder to mimic these soft labels, the student can achieve near-parity with the teacher—reaching an ROC-AUC of 0.9010 against the teacher's 0.9011—while being 400x faster.
Aligning the Vector Space
The goal of distillation in retrieval is to force the Bi-Encoder’s simple vector space to mirror the complex similarity distribution of the teacher. This is typically done through two primary mechanisms:
- KL-Divergence and Score Matching: The student is penalized if the probability distribution of its scores across a set of documents differs from the teacher's. Xingwei He and Yeyun Gong show in Metric-guided Distillation that this alignment allows the retriever to capture implicit connections that are usually only visible to high-precision rankers.
- Margin Distillation: Instead of matching exact scores, the student learns to match the difference in scores between a positive and a negative document. Thibault Formal and Carlos Lassance used this in SPLADE v2 to bring the power of Cross-Encoders to sparse, keyword-like indexes.
Distilling from Late Interaction and LLMs
The "teacher" doesn't always have to be a standard Cross-Encoder. Researchers are increasingly distilling knowledge from even more complex architectures:
- Late Interaction Models: Models like ColBERT use multiple vectors per document to allow for more granular matching. Keshav Santhanam and Omar Khattab showed in ColBERTv2 that "denoised supervision" from a teacher can significantly boost the quality of these multi-vector representations.
- Large Language Models (LLMs): In UDAPDR, Jon Saad-Falcon and team used LLMs to generate synthetic data and then distilled the resulting reranker's knowledge into a retriever. This allows a small model to inherit the "reasoning" capabilities of a giant model like GPT-4.
Why it Works: The Training Recipe Matters
A recurring theme across recent research is that the specific neural architecture (sparse vs. dense) is often less important than the training recipe.
In A Unified Framework for Learned Sparse Retrieval, Thong Nguyen and Andrew Yates argue that many performance gains attributed to new models actually come from distillation and hard-negative mining. By using a high-performance teacher to label data, we provide the student with a much "cleaner" signal. This is particularly vital in domain adaptation; as Nils Reimers demonstrates with Augmented SBERT, using a Cross-Encoder to label unlabeled pairs in a new domain can improve Bi-Encoder performance by up to 37 points.
Ultimately, distillation works because it allows the student to "see" the interaction it cannot compute. By mimicking the teacher's output, the Bi-Encoder learns to position its query and document embeddings such that their simple dot product approximates the complex, multi-layered attention of a Transformer.