Preserving Token Level Granularity with Multi Vector Retrieval

46 concepts45 papers1 proposal

When we compress a long text passage into a single fixed-length vector, we make a fundamental mathematical compromise. Standard dense retrieval architectures—such as dual encoders—typically pass a document through a transformer and extract a single vector representation, often via the [CLS] token or a mean-pooling operation.

This approach introduces a severe information bottleneck. By forcing an entire multi-sentence passage, a complex medical chart, or a multi-page document into a single point in a high-dimensional space, the fine-grained semantic nuances and localized facts are inevitably washed out. The geometric average of multiple distinct topics yields a blurry global representation that fails to match specific, targeted queries.

To overcome this limitation, a prominent research direction has emerged: moving away from monolithic single-vector representations toward multi-vector and late-interaction architectures that preserve token-level and structural granularity.

The Mechanics of Embedding Dilution

Why exactly does single-vector compression fail? The root of the problem lies in how pooling operations act as a semantic low-pass filter.

As early as sequence-to-sequence modeling, Karl Moritz Hermann et al. demonstrated in Teaching Machines to Read and Comprehend that encoding long documents into a single fixed-size vector creates a capacity bottleneck that discards localized details. In information retrieval, this manifests as embedding dilution. As shown by Ofer Egozi et al. in Concept-Based Feature Generation and Selection for Information Retrieval, representing long documents as single high-dimensional vectors causes specific, localized topics to be averaged out by the rest of the text.

This dilution introduces several distinct failure modes across different domains:

  1. Contrastive Conflicts during Training: When a single passage contains answers to multiple distinct questions, forcing the model to map the entire passage to a single vector causes optimization conflicts. Wu Hong et al. identify this in Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval, noting that single-vector compression lacks the granularity required to isolate specific evidence within a larger context.
  2. Loss of Structural Composition: Collapsing a document's multi-faceted word distributions into a single mean vector discards internal structural information. As argued by Charlotte Laclau et al. in Cross-lingual Document Retrieval using Regularized Wasserstein Distance, this reductionist approach prevents systems from distinguishing between documents that share similar average word meanings but possess entirely different semantic compositions.
  3. Domain-Specific Blind Spots: In specialized fields like medicine or scientific literature, a single global embedding frequently dilutes critical methodological details or niche technical terms. Man Luo et al. demonstrate this limitation in Improving Biomedical Information Retrieval with Neural Retrievers, explaining why traditional keyword matching often outperforms single-vector neural retrievers on dense, technical corpora. A similar effect is documented by Jeyun Lee et al. in Improving Scientific Document Retrieval with Academic Concept Index.

The Solution: Late Interaction and Multi-Vector Encodings

To bypass the single-vector bottleneck without incurring the massive computational cost of full cross-attention at retrieval time, researchers developed the late interaction paradigm.

The core intuition is simple: instead of aggregating token vectors before indexing, we preserve the individual vector for every token in the document. The interaction between the query and the document is deferred—hence "late interaction"—until the retrieval stage, where it is computed using lightweight vector operations.

Single-Vector (Dual Encoder):
Query ----> [Encoder] ----> Vector \
                                    +---> Dot Product ---> Score
Document -> [Encoder] -> Pooling -> Vector /

Multi-Vector (Late Interaction):
Query ----> [Encoder] ----> [ q1, q2, q3 ] \
                                            +---> MaxSim Alignment ---> Score
Document -> [Encoder] ----> [ d1, d2, d3 ] /

In Relevance-guided Supervision for OpenQA with ColBERT, Omar Khattab et al. formalized this via the MaxSim operator. For a query $Q$ and a document $D$, the relevance score is computed as the sum of the maximum dot products between each query token embedding and all document token embeddings:

$$\text{Score}(Q, D) = \sum_{q \in Q} \max_{d \in D} \left( q \cdot d^T \right)$$

This mechanism allows each query token to align dynamically with the single most relevant token in the document. If a document covers three different subjects, a query about the second subject will successfully align with the specific tokens representing that topic, completely unbothered by the presence of the other two subjects.

To make this paradigm practical at scale, Keshav Santhanam et al. introduced ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction, which employs a residual compression mechanism to slash the storage footprint of multi-vector representations by 6–10×. Further scaling and accessibility have been driven by frameworks like PyLate, developed by Raphaël Sourty in PyLate: Flexible Training and Retrieval for Late Interaction Models.

Alternative Granularity Scales

The push against single-vector representations is not restricted solely to token-level models; it spans a spectrum of granularities designed to balance storage efficiency and precision.

Sentence and Aspect-Level Decoupling

Instead of storing every token, some approaches segment documents into intermediate semantic units. Xixi Zhou et al. propose this in Matching Varying-Length Texts via Topic-Informed and Decoupled Sentence Embeddings to handle texts of wildly varying lengths without experiencing embedding dilution. Similarly, in multi-aspect sentiment analysis and document modeling, Nikolaos Pappas et al. utilize multiple instance learning in Explicit Document Modeling through Weighted Multiple-Instance Learning to assign distinct saliency weights to individual sentences, capturing nuanced information across different aspects that a global vector would obscure.

Visual Document Retrieval

The single-vector bottleneck becomes even more pronounced when documents contain rich visual layouts, charts, and figures. Compressing an entire document page into one vector completely destroys spatial relationships.

To solve this, Manuel Faysse et al. introduced ColPali in ColPali: Efficient Document Retrieval with Vision Language Models. By leveraging a Vision Language Model (VLM), ColPali avoids brittle text extraction pipelines entirely. It maps document pages directly as images, representing them as a collection of visual patch embeddings.

Using late interaction over these patches, queries can map directly to localized visual components on a page. The robustness of this visual multi-vector paradigm was subsequently validated by Jia-Huei Ju et al. in Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction.

Summary

The transition from single-vector pooling to multi-vector representations reflects a broader realization in information retrieval: compression should not happen before interaction. By preserving token, sentence, or patch-level granularities and leveraging late interaction, modern retrieval architectures successfully eliminate semantic blurring, ensuring that specific details remain discoverable no matter how large or complex the surrounding context.

Key Foundational Work

Concept-Based Feature Generation and Selection for Information Retrieval (2008) — contributed 1 concept elements to this direction.
Aggregating Continuous Word Embeddings for Information Retrieval (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_vector_representation_bottleneck__5f3395b8,
  title = {single-vector representation bottleneck (semantic granularity loss)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-vector-representation-bottleneck-semantic-granularity-loss/rdr_5f3395b862bc46c49afdf4ef4ea03fe2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_vector_representation_bottleneck__5f3395b8_v1,
  title = {single-vector representation bottleneck (semantic granularity loss)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-vector-representation-bottleneck-semantic-granularity-loss/rdr_5f3395b862bc46c49afdf4ef4ea03fe2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake