Distilling Cross-Encoders into Dual-Encoders for Efficient Retrieval

34 concepts29 papers1 proposal

Information retrieval systems face a fundamental trade-off between accuracy and latency. To find the most relevant documents for a given query, modern search engines typically rely on deep Transformer architectures. These models generally fall into two categories: Cross-Encoders and Dual-Encoders (also known as Bi-Encoders or Siamese networks).

A Cross-Encoder feeds the query and the document jointly into a neural network, allowing all-to-all self-attention across every token in both texts. This joint attention yields highly accurate relevance scores, but it comes with what Sebastian Hofstätter et al. term a massive "latency tax" in Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking. Because the query and document must be processed together, document representations cannot be precomputed. Running a heavy Transformer at query time for millions of candidate documents is computationally prohibitive for real-time production systems.

Conversely, a Dual-Encoder processes the query and the document independently, mapping each into a shared vector space. Because document embeddings can be precomputed and indexed offline, query-time scoring reduces to a lightning-fast vector dot product or a sparse table lookup. However, because the query and document embeddings do not interact at a token level during encoding, Dual-Encoders lose fine-grained semantic nuances, resulting in a significant drop in ranking accuracy.

Cross-stage knowledge distillation resolves this dilemma. Instead of choosing between speed and accuracy, researchers use a computationally expensive Cross-Encoder as a "teacher" to guide the training of a highly efficient "student" retriever (such as a Dual-Encoder, a sparse retriever, or a generative model). By forcing the student to mimic the nuanced scoring patterns of the teacher, the complex token-to-token interactions of joint attention are effectively baked into the student's simplified representation space. At inference time, the teacher is discarded, leaving an ultra-low-latency retriever that performs with near-Cross-Encoder accuracy.


How Knowledge Transfers Across Architectures

The core intuition behind cross-encoder distillation is that hard binary labels (e.g., a document is either "relevant" or "irrelevant") are too sparse to teach a dual-encoder the complex semantic landscape mapped by a cross-encoder. Instead, the student model is trained on the continuous, soft outputs generated by the teacher.

Soft Labels and Probability Distributions

When a Cross-Encoder evaluates a set of candidate documents for a query, it outputs a continuous spectrum of relevance scores. Training a student model to match these scores allows it to learn not just which documents are relevant, but how much more relevant one document is relative to another.

Matej Kocián et al. demonstrate this in Siamese BERT-Based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset, showing that training a Siamese bi-encoder on soft labels generated by a cross-encoder effectively closes the performance gap between the two architectures. Similarly, Jiri Gesi et al. leverage soft-label training in Behavior-driven Query Similarity Prediction based on Pre-trained Language Models for E-Commerce Search to capture deep interaction signals while maintaining the low latency required for real-time production environments.

Margin-Based Distillation

Rather than matching absolute scores, a student can be optimized to match the relative difference (or margin) between pairs of documents as scored by the teacher. For instance, if the teacher scores Document A as a 9.5 and Document B as a 4.0, the margin is 5.5. The student is penalized if its independent vector dot products do not reflect that same relative gap.

This approach is widely implemented using a margin-based Mean Squared Error (MSE) loss. Arnaud Sors et al. utilize this mechanism in Naver Labs Europe (SPLADE) @ TREC Deep Learning 2021 (TREC 2021) to optimize first-stage sparse retrieval models. By minimizing the margin MSE between the sparse retriever and a heavy BERT-based re-ranker, the efficient first-stage model learns to approximate nuanced relevance scoring.

The authors of Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning, Che Chen et al., also employ a margin-based loss to transfer complex ranking logic into bi-encoders, allowing them to adapt to specialized target domains without human-labeled data.

Pairwise and Listwise Alignment

Beyond pointwise and margin matching, distillation can optimize the global ranking order. The authors of PairDistill: Pairwise Relevance Distillation for Dense Retrieval focus explicitly on relative pairwise relevance, teaching the retriever to understand fine-grained comparisons between documents.

In multi-stage recommendation and search pipelines, Xu Huang et al. introduce the Cooperative Retriever and Ranker in Deep Recommenders (CoRR) framework, which uses a sampled KL divergence loss. This treats the ranker's top outputs as a soft probability distribution, training the retriever to prioritize items that the ranker would likely score highly, thereby aligning the candidate generation phase with the final scoring phase.


Aligning Vector Geometry with Joint Attention

Why does this distillation process actually work? Geometrically, a dual-encoder must arrange its independent query and document embeddings such that their spatial proximity reflects their semantic alignment. Distillation forces the student to warp its embedding space to mirror the cross-encoder's attention landscape.

Cross-Encoder (Teacher)           Dual-Encoder (Student)
[Query + Doc] Joint Attention     [Query] -> Vector ──┐
             │                                        ├──> Dot Product (Mimics Teacher)
             ▼                                        │
        Soft Score  ───────────────────────────────> [Doc]   -> Vector ──┘

To achieve this, Yanmeng Wang et al. introduce a Geometry Alignment Mechanism in Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval. This framework explicitly forces a dual-encoder to mimic the sophisticated embedding space of a cross-encoder during training, transferring knowledge from joint-attention models to independent-encoding models.

To capture token-level nuances within a single vector, single-author work by Xingyi Cheng (Dual-View Distilled BERT for Sentence Embedding (arxiv 2021)) injects word-level interaction knowledge into Siamese architectures. This "dual-view" distillation allows the bi-encoder to retain fine-grained semantic nuances without increasing inference-time complexity.

Similarly, Yi Liu et al. demonstrate in Towards Better Entity Linking with Multi-View Enhanced Distillation that a cross-encoder teacher can identify exactly which specific segments of a text description are most relevant to a given context. The dual-encoder student learns from these multi-view signals, capturing fine-grained matching patterns it would otherwise miss.

This geometric alignment can also be structured around the documents themselves. Ruiyang Ren et al. propose the PAIR framework in PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval. PAIR enforces a passage-centric constraint: a positive passage must not only be close to its query, but it must be significantly closer to that query than to any "hard" negative queries. By using a cross-encoder to generate high-quality pseudo-labels for unlabeled query-passage pairs, the dual-encoder learns an optimized, contrastive embedding structure.


Scaling Up with Intermediate Teachers

While cross-encoders make excellent teachers, calculating cross-attention scores for millions of query-document pairs during training is incredibly expensive. To scale distillation, researchers often employ intermediate architectures or multi-stage distillation pipelines.

Late-Interaction Models as Teachers

Late-interaction architectures (such as ColBERT) serve as an effective middle ground. They encode queries and documents independently into matrices of token-level embeddings and defer token interaction to a lightweight "MaxSim" operator at the very end.

Sheng-Chieh Lin et al. exploit this property in In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval. Because late-interaction models are significantly faster than cross-encoders, they can be used as distillation teachers over massive sets of contrastive examples. This allows the student bi-encoder to learn from a much larger pool of in-batch negatives during training without exceeding hardware memory limits.

Distilling into Sparse and Adapter Modules

Late-interaction models can also serve as the target for further distillation. Stéphane Clinchant et al. introduce SPLATE in SPLATE: Sparse Late Interaction Retrieval, which distills a frozen late-interaction model into a lightweight sparse retrieval adapter. This ensures that the fast candidate generation phase shares the exact same semantic logic as the final late-interaction ranker, reducing the number of documents that must be re-ranked.

When adapting to completely new domains, Jon Saad-Falcon et al. propose UDAPDR in UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers. The framework uses large language models to generate synthetic data, trains high-accuracy cross-encoder rerankers on that data, and then distills their knowledge into a late-interaction retriever. This pipeline allows the retriever to approximate complex cross-encoder relevance modeling while maintaining millisecond-level response times via pre-computed document embeddings.


Extending Beyond Dense Vectors

The principles of cross-encoder distillation extend far beyond traditional dense bi-encoders. The same paradigm is actively used to optimize learned sparse representations, generative retrieval models, and large language model (LLM) rankers.

  • Learned Sparse Representations: Instead of dense vectors, models like SPLADE map text to high-dimensional, sparse lexical vectors. Franco Maria Nardini et al. show in Effective Inference-Free Retrieval for Learned Sparse Representations that distilling knowledge from a cross-encoder teacher allows sparse retrieval models to inherit sophisticated relevance signals. This enables inference-free retrieval via fast table lookups while maintaining high accuracy.
  • Generative Retrieval: Generative retrievers map queries directly to document identifiers (docIDs) using an autoregressive sequence-to-sequence model. Yongqi Li et al. introduce Distillation Enhanced Generative Retrieval, which distills the ranking expertise of cross-encoders into a generative student via a distilled RankNet loss, bypassing the computational overhead of cross-attention during inference.
  • Encoder-Decoder Optimization: In the context of embedding generation, Jianmo Ni et al. explore the scaling dynamics of text-to-text models in Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models. They highlight that transferring the superior semantic performance of heavy encoder-decoder pooling architectures into efficient encoder-only models via distillation represents a vital optimization path for low-latency applications.

Real-World Applications and Domain Adaptation

Cross-stage distillation is particularly powerful when deploying models to specialized domains or production environments where human annotations are scarce or inference budgets are strictly capped.

Zero-Shot Domain Adaptation

In zero-shot settings, a cross-encoder can act as an automated labeler to bootstrap a faster retrieval model. For entity linking across millions of Wikipedia pages, Ledell Wu et al. develop BLINK in Scalable Zero-shot Entity Linking with Dense Entity Retrieval. BLINK distills a deep cross-encoder re-ranker into a fast bi-encoder candidate generator, allowing the system to accurately scan millions of entities in milliseconds.

For highly technical domains like medicine, Jing Lu et al. demonstrate in Zero-shot Hybrid Retrieval and Reranking Models for Biomedical Literature that distilling a cross-attention reranker into a dual-encoder retriever yields top-tier search performance without requiring domain-specific human annotations.

Production Constraints and Contextual Retrieval

In industrial applications like digital advertising, Zhipeng Jin et al. show in Enhancing Dynamic Image Advertising with Vision-Language Pre-training that distilling a complex multi-modal relevance model into a faster retrieval model allows large-scale candidate selection systems to operate within strict real-time latency budgets.

When dealing with long documents or listwise context, Qi Liu et al. propose PE-Rank in Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models. PE-Rank uses a distillation loss to align a compressed, embedding-based model with a teacher LLM that has access to full-text passages, achieving a 4.5x speedup in listwise reranking.

Distilling Feedback and Rewards

The distillation framework also adapts well to optimizing models based on downstream feedback or complex reward functions:

Key Foundational Work

In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021) — contributed 2 concept elements to this direction.
Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2020) — contributed 1 concept elements to this direction.
Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_encoder_distillation_ee1382fa,
  title = {cross-encoder distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-encoder-distillation/rdr_ee1382fa7d404b829bcb402eba32a49a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_encoder_distillation_ee1382fa_v1,
  title = {cross-encoder distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-encoder-distillation/rdr_ee1382fa7d404b829bcb402eba32a49a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake