Scaling Multi-Vector Retrieval via Efficient Storage

52 concepts47 papers

In modern information retrieval, a fundamental tension exists between representational richness and system scalability. Traditional "dense retrieval" models, or bi-encoders, compress an entire document into a single fixed-length vector. While efficient, Zhang et al. et al. argue this creates a "semantic bottleneck," where a single vector struggles to capture the diverse facets of a multi-topic document.

To solve this, researchers introduced multi-vector retrieval, most notably through the "late interaction" paradigm. Instead of squashing a document into one vector, models like ColBERT, proposed by Khattab and Zaharia, store an embedding for every token in a document. During search, the query also becomes a set of vectors, and the system calculates the maximum similarity between each query token and all document tokens. This granular matching significantly improves accuracy, but it introduces a massive "storage tax": an index that can be 10x to 100x larger than single-vector alternatives.

The research direction explored here focuses on mitigating this storage overhead to make high-precision multi-vector retrieval practical for production environments.

The Storage Bottleneck

The primary challenge of multi-vector models is linear scaling relative to document length. As Luan et al. observe, representing documents with multiple vectors increases the memory footprint and RAM requirements to levels that make massive datasets difficult to manage. For instance, Khattab and Zaharia noted that even with dimensionality reduction, storing contextualized embeddings for every token can require tens of gigabytes for relatively small datasets.

This overhead is even more pronounced in Visual Document Retrieval (VDR). Models like ColPali, reproduced by Ju et al., treat document pages as images and store embeddings for every visual patch. Faysse et al. point out that while this avoids brittle OCR processes, it creates a critical tradeoff between the performance of patch-level representations and the infrastructure costs of maintaining large-scale vector indices.

Pruning the Redundant Tail

One strategy to reduce overhead is identifying which vectors are actually necessary for retrieval. Liu et al. demonstrate that late-interaction models behave surprisingly like traditional keyword-based systems, relying heavily on a few "important" words. By applying simple pruning heuristics—such as keeping only high-importance words or the first 75% of a document—storage can be significantly reduced with negligible impact on quality.

More sophisticated approaches involve "pseudo-query" extraction. Kim et al. developed PQA-ColBERT, which uses a specialized extractor to identify the most important terms at indexing time. This bridges the gap between massive multi-vector models and fast single-vector retrievers, slashing memory usage by 3x. Similarly, Kang et al. propose modeling documents as a distribution of potential queries, allowing the system to capture semantic breadth without storing every individual token vector.

Compression and Sparse Routing

Beyond pruning, researchers have developed advanced compression techniques to shrink the size of each stored vector. ColBERTv2, introduced by Santhanam et al., employs residual compression. Instead of storing full high-dimensional vectors, it stores a codebook of centroids and only records the "residual" (the difference) between a token and its nearest centroid. This allows multi-vector retrieval to become nearly as space-efficient as single-vector models.

Another approach involves Dynamic Lexical Routing. Li et al. introduced CITADEL, which routes query and document tokens to specific lexical keys. This avoids the need for all-to-all comparisons, making the system 40x faster. Similarly, Lin et al. developed SLIM, which projects neural embeddings into a sparse lexical space compatible with traditional inverted indexes like Lucene, reducing the index size by 40% compared to standard ColBERT.

Hardware-Aware Optimization

Finally, recent work focuses on the low-level efficiency of the retrieval engine itself. Scheerer et al. developed WARP, an engine that uses implicit decompression and dynamic similarity imputation to achieve a 41x speedup over reference implementations. By optimizing how vectors are stored and accessed in memory, these engines allow multi-vector models to move from academic benchmarks into real-time, large-scale production environments.

As Nardini shows with EMVB, leveraging bit-vector pre-filtering can further reduce memory footprints by 1.8x, proving that the future of high-fidelity retrieval lies in the clever intersection of neural representation and classical data compression.

Key Foundational Work

Learning K-way D-dimensional Discrete Embedding for Hierarchical Data Visualization and Retrieval (2019) — contributed 1 concept elements to this direction.
Beyond Vector Spaces: Compact Data Representation as Differentiable Weighted Graphs (2019) — contributed 1 concept elements to this direction.
Combining document representations for known-item search (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_vector_storage_overhead_c77e0352,
  title = {multi-vector storage overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-vector-storage-overhead/rdr_c77e035275014c47ac4cbda450208551},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_vector_storage_overhead_c77e0352_v1,
  title = {multi-vector storage overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-vector-storage-overhead/rdr_c77e035275014c47ac4cbda450208551/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake