PLAID: An Efficient Engine for Late Interaction Retrieval

Keshav Santhanam, Omar Khattab, Christopher Potts, Matei Zaharia

Abstract

Late interaction models like ColBERTv2 have established a new state-of-the-art in Information Retrieval (IR) by enabling fine-grained token-level matching. However, the computational overhead of these models has historically limited their deployment in production environments. This paper presents PLAID (Performance-optimized Late Interaction Driver), a high-performance engine designed to accelerate late interaction retrieval. By employing a multi-stage filtering pipeline and hardware-optimized C++ kernels, PLAID achieves up to a 45x speedup on CPUs and a 7x speedup on GPUs compared to existing implementations, maintaining peak retrieval quality even at scales exceeding 140 million documents.

Introduction

Modern Information Retrieval has evolved from simple keyword-based matching (e.g., BM25) to sophisticated neural ranking models. Among these, "late interaction" models such as ColBERT have emerged as a superior alternative to "single-vector" models like DPR. While single-vector models compress an entire document into a single embedding, late interaction models represent documents as collections of token-level vectors. This allows the model to perform "MaxSim" operations—aligning query tokens with the most relevant document tokens—to capture nuanced semantic relationships.

Despite their accuracy, late interaction models are computationally demanding. The process of searching through massive vector indices is often bottlenecked by slow disk lookups, decompression overhead, and the sheer volume of matrix operations required for scoring. PLAID addresses these bottlenecks, transforming late interaction from a research curiosity into a production-ready technology.

The PLAID Engine: A Multi-Stage Sieve

PLAID treats the retrieval process as a multi-stage "sieve," progressively narrowing down the search space using increasingly granular representations of the data. This architecture is built on several key innovations:

1. Centroid-Based Interaction

ColBERTv2 utilizes clustering to compress vectors into centroids. PLAID leverages these centroids as "lightweight" proxies for the actual document content. By representing passages as bags of centroids, the engine can estimate the relevance of a document without loading its full, high-dimensional residual data.

2. The Multi-Stage Pipeline

The retrieval process is divided into four distinct stages:

  • Stage 1 (Candidate Generation): Identifies a broad set of potential matches using centroid-based indexing.
  • Stages 2 & 3 (Refinement): Uses centroid interaction to score and prune the candidate list. This allows the system to discard low-scoring passages early in the pipeline.
  • Stage 4 (Full Scoring): Performs full residual decompression and MaxSim scoring only on the tiny fraction of documents that survive the previous stages.

3. Hardware-Level Optimization

To overcome the limitations of generic deep learning frameworks, PLAID introduces custom C++ kernels. These kernels are specifically optimized for "padding-free" MaxSim operations. By handling "ragged" tensors (passages of varying lengths) without artificial padding, the engine significantly reduces memory bandwidth requirements and compute cycles. Additionally, PLAID utilizes lookup-table-based decompression to accelerate the retrieval of vector residuals.

Experimental Evaluation

The authors evaluated PLAID across several large-scale benchmarks, including MS MARCO v1 (8.8M passages), Wikipedia (NQ), and LoTTE (long-tail topics). To test the engine's scalability, it was also deployed against MS MARCO v2, which contains 138.4 million passages.

Performance and Latency

PLAID demonstrates transformative performance gains:

  • Speed: It is 2.5–7× faster on GPUs and 9–45× faster on CPUs compared to the original ColBERTv2 implementation.
  • Latency: On a standard CPU, PLAID can search 140 million passages in a few hundred milliseconds. On a GPU, this latency drops to tens of milliseconds.
  • Efficiency: PLAID achieves latency competitive with sparse models like SPLADEv2 while maintaining the superior accuracy characteristic of late interaction.

Retrieval Quality

Crucially, these speedups do not come at the cost of accuracy. In empirical tests, PLAID matched the Mean Reciprocal Rank (MRR@10) of the original ColBERTv2 model exactly. In some out-of-domain scenarios, the pruning strategies even led to slight improvements in retrieval quality.

Discussion and Insights

A primary insight of this work is the effectiveness of centroids as proxies for full vectors. The authors found that retrieving the top 10 documents using only centroid data has a 99%+ probability of including the same documents that the full, uncompressed model would select. This supports the "Power Law" of relevance in IR: only a small subset of token interactions (the "tail") contributes significantly to a query's final score. By pruning unimportant centroids early, PLAID discards the vast majority of the search space with negligible loss in fidelity.

Conclusion and Future Work

PLAID successfully bridges the gap between high-accuracy late interaction models and the latency requirements of production search engines. While the current implementation is highly optimized, future work remains in achieving better load balancing across CPU threads and developing custom padding-free kernels for GPU stages that still rely on standard frameworks. Nevertheless, PLAID establishes a new benchmark for efficient, large-scale neural retrieval.

Summary updated

BibTeX
@inproceedings{santhanam2022plaid,
  author = {Keshav Santhanam and Omar Khattab and Christopher Potts and Matei Zaharia},
  title = {{PLAID: An Efficient Engine for Late Interaction Retrieval}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2022},
  doi = {10.1145/3511808.3557325},
}
Figure