TL;DR
This paper challenges the popular industry narrative that modern AI search requires a dedicated, specialized "vector database." By combining OpenAI’s embedding APIs with the venerable Apache Lucene search library, the authors demonstrate that existing, widely deployed infrastructure can achieve state-of-the-art retrieval performance without the added complexity of a new software stack.
What Problem Does This Paper Address?
The rise of Large Language Models (LLMs) has popularized "dense retrieval," where queries and documents are converted into high-dimensional vectors (embeddings) and matched based on similarity. Currently, a "cottage industry" of startups argues that managing these vectors requires a dedicated "vector store" (e.g., Pinecone, Weaviate, Milvus).
The authors argue that search is a brownfield application—meaning most organizations already have massive investments in existing search infrastructure, primarily built on Apache Lucene (the engine behind Elasticsearch, OpenSearch, and Solr). They set out to prove that Lucene’s built-in capabilities are "adequate" for modern AI search, making the introduction of a separate vector database an unnecessary architectural burden for most enterprises.
Methods
The researchers implemented a standard bi-encoder architecture:
- Embedding Generation: They used OpenAI’s
text-embedding-ada-002(ada2) model via API to generate 1536-dimensional vectors for both queries and documents. - Indexing and Retrieval: They used Lucene 9.5.0 (via the Anserini toolkit). Lucene utilizes Hierarchical Navigable Small-World (HNSW) graphs, which are the industry standard for efficient approximate nearest neighbor (ANN) search.
- The "Janky" Hack: At the time of the study, Lucene had a hard-coded limit of 1024 dimensions for vectors. Since OpenAI’s vectors are 1536 dimensions, the authors had to manually patch the Lucene source code to support the larger size—a limitation that has since been addressed in the main Lucene codebase.
Datasets
The authors evaluated their approach using the MS MARCO passage ranking collection, which contains approximately 8.8 million passages. They tested performance across three benchmarks:
- MS MARCO development set queries.
- TREC 2019 Deep Learning Track.
- TREC 2020 Deep Learning Track.
Key Insights and Ideas
- Capabilities vs. Components: The authors distinguish between the need for vector search capabilities and the need for a new software component. You can have the former without the latter.
- The "Free Lunch" of Existing Tech: If an organization already uses Elasticsearch or Solr, adding vector search is an incremental update rather than a "rip and replace" operation.
- API-First Simplicity: By using OpenAI's embedding API, the most computationally expensive part of the process (inference) is offloaded. This leaves Lucene to do what it has done for decades: efficiently index and retrieve data.
- Performance Trade-offs: While specialized libraries like Faiss are faster than Lucene in raw throughput, Lucene scales better with multiple threads and integrates seamlessly with traditional keyword search (BM25), which is often necessary for "hybrid" search strategies.
Key Results
The "OpenAI + Lucene" combo proved to be highly competitive with specialized, state-of-the-art dense retrieval models:
- Effectiveness: On the MS MARCO dev set, the approach achieved an RR@10 of 0.343, performing similarly to established models like TAS-B (0.340) and TCT-ColBERTv2 (0.358).
- Efficiency: On a dated high-end server (from 2017), the system achieved 9.8 queries per second (QPS) while fetching 1,000 hits per query.
- Storage: The raw 1536-dimensional vectors for 8.8 million passages consumed about 54 GB of disk space (stored as 32-bit floats).
Limitations and Future Work
- The "Jank" Factor: The authors admit their demonstration required "janky" implementation tricks to bypass Lucene's then-current dimension limits. While these features are now in the Lucene "trunk," they weren't in an official release at the time of writing.
- Raw Speed: Lucene still lags behind specialized libraries like Faiss in terms of indexing speed and query latency. However, the authors argue this gap is narrowing and is a "small matter of programming" (SMOP) rather than a fundamental flaw.
- Managed Services: The paper briefly notes that fully managed services like Vespa offer an alternative path, but questions whether enterprises are ready to make the leap to a completely new managed ecosystem versus sticking with their current Lucene-based tools.
Summary updated