Vector Search with OpenAI Embeddings: Lucene Is All You Need

Jimmy Lin, Ronak Pradeep, Tommaso Teofili, Jasper Xian

Abstract

We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection. The main goal of our work is to challenge the prevailing narrative that a dedicated vector store is necessary to take advantage of recent advances in deep neural networks as applied to search. Quite the contrary, we show that hierarchical navigable small-world network (HNSW) indexes in Lucene are adequate to provide vector search capabilities in a standard bi-encoder architecture. This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern "AI stack" for search, since such applications have already received substantial investments in existing, widely deployed infrastructure.

Introduction

The rise of Large Language Models (LLMs) has popularized "dense retrieval," where queries and documents are converted into high-dimensional vectors (embeddings) and matched based on similarity. Currently, a "cottage industry" of startups argues that managing these vectors requires a dedicated "vector store" (e.g., Pinecone, Weaviate, Milvus).

This paper addresses the problem of architectural bloat in search applications. We argue that search is a brownfield application—meaning most organizations already have massive investments in existing search infrastructure, primarily built on Apache Lucene (the engine behind Elasticsearch, OpenSearch, and Solr). We set out to prove that Lucene’s built-in capabilities are "adequate" for modern AI search, making the introduction of a separate vector database an unnecessary burden for most enterprises.

Methodology

We implemented a standard bi-encoder architecture to evaluate the feasibility of using Lucene for dense retrieval. The process involves two primary stages: encoding and retrieval.

  1. Embedding Generation: We utilized OpenAI’s text-embedding-ada-002 (ada2) model via API to generate 1536-dimensional vectors for both queries and documents.
  2. Indexing and Retrieval: We used Lucene 9.5.0 (via the Anserini toolkit). Lucene utilizes Hierarchical Navigable Small-World (HNSW) graphs, which are the industry standard for efficient approximate nearest neighbor (ANN) search.
  3. Implementation Details: At the time of the study, Lucene had a hard-coded limit of 1024 dimensions for vectors. To accommodate OpenAI’s 1536-dimensional vectors, we manually patched the Lucene source code—a limitation that has since been addressed in the main Lucene codebase.
Standard bi-encoder architecture for dense retrieval.

Datasets

Performance was evaluated using the MS MARCO passage ranking collection, containing approximately 8.8 million passages. We tested across three benchmarks: the MS MARCO development set, and the TREC 2019 and 2020 Deep Learning Track query sets.

Key Results and Insights

The "OpenAI + Lucene" combination proved to be highly competitive with specialized, state-of-the-art dense retrieval models.

  • Effectiveness: On the MS MARCO dev set, the approach achieved an RR@10 of 0.343, performing similarly to established models like TAS-B (0.340) and TCT-ColBERTv2 (0.358). As shown in the performance table below, the OpenAI ada2 model on Lucene holds its own against both learned sparse and dense representations.
  • Efficiency: On a 2017-era high-end server, the system achieved 9.8 queries per second (QPS) while fetching 1,000 hits per query.
  • Storage: The raw 1536-dimensional vectors for 8.8 million passages consumed approximately 54 GB of disk space (stored as 32-bit floats).
Retrieval effectiveness comparison across MS MARCO and TREC benchmarks.

Our findings highlight several key ideas:

  • Capabilities vs. Components: There is a distinction between the need for vector search capabilities and the need for a new software component. Organizations can achieve the former without the latter.
  • The "Free Lunch" of Existing Tech: For organizations already using Elasticsearch or Solr, adding vector search is an incremental update rather than a "rip and replace" operation.
  • API-First Simplicity: By using OpenAI's embedding API, the most computationally expensive part of the process (inference) is offloaded, allowing Lucene to focus on its core strength: efficient indexing and retrieval.

Limitations and Future Work

While Lucene is effective, it is not without limitations. Our demonstration required custom patches to bypass dimension limits, though these are now integrated into the Lucene "trunk." Furthermore, while Lucene's raw throughput lags behind specialized libraries like Faiss, we believe this gap is narrowing and represents a "small matter of programming" rather than a fundamental architectural flaw.

Future work will explore the integration of these capabilities into fully managed services and the continued optimization of HNSW performance within the Lucene ecosystem. For now, the evidence suggests that for most enterprises, Lucene is indeed all you need.

Summary updated

Abstract

We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection. The main goal of our work is to challenge the prevailing narrative that a dedicated vector store is necessary to take advantage of recent advances in deep neural networks as applied to search. Quite the contrary, we show that hierarchical navigable small-world network (HNSW) indexes in Lucene are adequate to provide vector search capabilities in a standard bi-encoder architecture. This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern "AI stack" for search, since such applications have already received substantial investments in existing, widely deployed infrastructure.
BibTeX
@inproceedings{lin2024vector,
  author = {Jimmy Lin and Ronak Pradeep and Tommaso Teofili and Jasper Xian},
  title = {{Vector Search with OpenAI Embeddings: Lucene Is All You Need}},
  booktitle = {WSDM},
  year = {2024},
  doi = {10.1145/3616855.3635691},
}
Figure