Semantic Search via Dense Vector Retrieval

199 concepts192 papers2 proposals

For decades, information retrieval relied on a simple assumption: if a document is relevant to a query, they must share words. Lexical frameworks like BM25 score relevance based on exact keyword matches and term frequencies. However, this approach breaks down when facing the nuances of human language. A user searching for "feline care" might completely miss a highly relevant document titled "How to Feed Your Cat" because there is zero literal keyword overlap.

Dense vector retrieval solves this problem by shifting the search paradigm from lexical matching to geometric proximity. Instead of matching strings, it maps both queries and documents into a shared, high-dimensional continuous vector space where spatial closeness corresponds directly to semantic similarity.

The Core Mechanism: The Dual-Encoder Architecture

To understand why dense retrieval works, we must look at how these high-dimensional spaces are constructed. The standard architecture relies on a dual-encoder (or bi-encoder) framework.

Query ------> [ Query Encoder ] -------> Vector Q \
                                                   +---> Dot Product / Cosine Similarity ---> Relevance Score
Document ---> [ Document Encoder ] ----> Vector D /

In this setup, two independent neural networks (often based on Transformer architectures like BERT) process the inputs:

  1. One encoder compresses the user's query into a single fixed-length vector $\mathbf{q}$.
  2. A separate encoder compresses the target document or entity into a vector $\mathbf{d}$ of the same dimensionality.

Because these encoders are trained jointly using contrastive learning, they learn to project conceptually related inputs to nearby coordinates in the vector space, regardless of the specific words used. The final relevance score is computed using simple vector algebra, such as the dot product ($\mathbf{q} \cdot \mathbf{d}$) or cosine similarity.

As explored by Hamed Zamani et al. in their framework for Retrieval-Enhanced Machine Learning, this design decouples parametric reasoning from non-parametric memory. The model generates high-dimensional vector embeddings that represent its specific information needs, allowing the retrieval system to scan massive databases without requiring the underlying facts to be baked directly into the model's weights.

Balancing Speed and Accuracy: Two-Stage Retrieval

While computing the dot product between two vectors is computationally cheap, doing it millions of times for every single user query becomes a bottleneck. To make dense retrieval scalable, systems typically employ a two-stage architecture.

In Scalable Zero-shot Entity Linking with Dense Entity Retrieval, Ledell Wu et al. demonstrated this trade-off using a system called BLINK.

  • Stage 1: The Bi-Encoder. The system uses a fast bi-encoder combined with Approximate Nearest Neighbor (ANN) search algorithms to instantly prune a database of millions of candidates down to a small pool of candidate matches (e.g., the top 100). Because the document vectors can be pre-computed and indexed offline, this stage takes only milliseconds.
  • Stage 2: The Cross-Encoder. Once the candidate pool is small, a deeper, more computationally expensive "cross-encoder" takes over. The cross-encoder feeds the query and the candidate document into a single network simultaneously, allowing full token-to-token attention across both texts. This yields highly precise re-ranking scores but is only feasible because the first-stage dense retrieval narrowed down the search space.

Bridging Modalities and Formats

One of the most powerful properties of dense vector spaces is their ability to act as a universal translation layer. Because vectors are just coordinates, entirely different data formats can be mapped into the same space as long as their underlying concepts align.

Cross-Lingual Retrieval

In traditional search, finding a Persian document using an English query requires explicit machine translation. Dense retrieval bypasses this. As highlighted in the Overview of the TREC 2024 NeuCLIR Track, training embedding models on multilingual corpora allows them to map queries and documents from different languages into a language-agnostic semantic space. An English query vector will naturally land near a Russian or Persian document vector if they describe the same concept.

Text-to-Structured Data

This alignment property extends to structured data. For instance, Siva Reddy et al. introduced The StatCan Dialogue Dataset, which uses dense passage retrieval to map natural language conversational turns and structured table metadata into a shared vector space. This allows users to discover complex data tables based on conversational intent rather than rigid schema matching.

Beyond Text: Biology and Neuroscience

Because neural encoders can be trained on any sequential or structured data, dense retrieval has expanded far beyond natural language processing:

  • Proteomics: Zhigang Hu et al. developed the Dense Homolog Retriever in Fast, sensitive detection of protein homologs using deep dense retrieval. By replacing character-matching sequence alignments with vector similarity measures, their system identifies remote protein homologs that share structural or functional traits despite having entirely different sequence identities.
  • Neuroscience: In Brain2Music: Reconstructing Music from Human Brain Activity, Timo I. Denk et al. used brain-predicted embeddings from fMRI scans to query a large-scale music database. By finding the closest matches in a shared vector space, they successfully retrieved existing musical tracks that matched the genre and mood of what a human subject was actively hearing.

Optimizing Systems via Semantic Caching

The flexibility of dense retrieval also provides massive efficiency gains at the system infrastructure level. In modern large language model (LLM) deployments, exact-match caching is notoriously inefficient. If a user asks "How do I fix a leaky pipe?" and another asks "What is the method to repair a pipe that leaks?", an exact-match cache treats them as entirely separate requests, executing expensive model inferences twice.

To solve this, Banghua Zhu et al. proposed a framework for semantic caching in Towards Optimal Caching and Model Selection for Large Model Inference. By converting incoming queries into dense embeddings, the system can calculate the cosine similarity against a cache of previously answered questions. If a new query lands within a tight geometric radius of a past query, the system instantly serves the cached result, drastically reducing latency and compute costs.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_embedding_based_retrieval_b281a57a,
  title = {embedding-based retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-based-retrieval/rdr_b281a57afff640d1a8e74116e00d5b6d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_embedding_based_retrieval_b281a57a_v1,
  title = {embedding-based retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-based-retrieval/rdr_b281a57afff640d1a8e74116e00d5b6d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake