Operational Advice for Dense and Sparse Retrievers: HNSW, Flat, or Inverted Indexes?

Jimmy Lin

Abstract

Practitioners working on dense retrieval today face a bewildering number of choices. Beyond selecting the embedding model, another consequential choice is the actual implementation of nearest-neighbor vector search. While best practices recommend HNSW indexes, flat vector indexes with brute-force search represent another viable option, particularly for smaller corpora and for rapid prototyping. In this paper, we provide experimental results on the BEIR dataset using the open-source Lucene search library that explicate the tradeoffs between HNSW and flat indexes (including quantized variants) from the perspectives of indexing time, query evaluation performance, and retrieval quality. With additional comparisons between dense and sparse retrievers, our results provide guidance for today's search practitioner in understanding the design space of dense and sparse retrievers. To our knowledge, we are the first to provide operational advice supported by empirical experiments in this regard.

Introduction

Building Retrieval-Augmented Generation (RAG) systems involves navigating a complex landscape of implementation choices. While academic research often focuses on marginal gains in accuracy, practitioners require empirical data on the operational trade-offs between different indexing strategies. This work, authored by Jimmy Lin, provides a "reality check" by benchmarking HNSW, flat, and inverted indexes within the Lucene search library, offering concrete guidance for real-world deployment.

Methodology and Experimental Setup

The study utilizes the BEIR (Benchmarking IR) dataset, categorized into three tiers based on corpus size ($|C|$):

  • Small: < 100k documents (e.g., NFCorpus, SciFact).
  • Medium: 100k to 1M documents (e.g., TREC-COVID, Quora).
  • Large: > 1M documents (e.g., Natural Questions, BioASQ).

Retrieval performance was evaluated using the BGE (bge-base-en-v1.5) model for dense retrieval and SPLADE++ EnsembleDistil for sparse retrieval, with BM25 as a baseline. Index types compared include HNSW (graph-based), Flat (brute-force), and Inverted indexes. Experiments were conducted on a 2022 Mac Studio (M1 Ultra) with 16 threads, measuring both cached query performance and on-the-fly encoding via ONNX.

Key Findings

1. The "Small Corpus" Threshold

For datasets with fewer than 100,000 documents, the performance gap between HNSW and flat indexes is negligible. In these cases, flat indexes are often preferable due to their simplicity and the fact that they provide exact, rather than approximate, results. This makes them ideal for rapid prototyping and exploratory phases.

2. The Indexing vs. Query Performance Trade-off

While HNSW excels at query time for large datasets, it carries a significant "indexing tax." For instance, on the 15M-document BioASQ dataset, HNSW required over 76 minutes to index, whereas a flat index took only 10 minutes. However, at query time, HNSW achieved ~56 QPS (cached) compared to just 2.6 QPS for the flat index, demonstrating its necessity for large-scale production environments.

Comprehensive comparison of retrieval effectiveness (nDCG@10) and efficiency (QPS) across various BEIR datasets and index types.

3. Quantization: An Efficiency "Free Lunch"

The study highlights that int8 quantization is highly effective. Applying quantization to dense vectors significantly boosts Queries Per Second (QPS)—by as much as 140% in some cases—with a minimal impact on retrieval quality. For the BioASQ dataset, the combined quality drop from HNSW and quantization was only ~4% relative to an exact flat index.

Impact of int8 quantization on HNSW index performance, showing substantial QPS gains with minimal loss in nDCG@10.

4. Dense vs. Sparse Retrieval

Interestingly, dense retrieval using HNSW and sparse retrieval using SPLADE show comparable query latency. There is no clear efficiency winner between the two; the choice should be driven by domain-specific requirements rather than raw performance metrics. BM25, however, remains the speed champion, being nearly an order of magnitude faster than neural methods.

Operational Advice for Practitioners

  • For corpora < 100k docs: Start with flat indexes for simplicity and exactness.
  • For large-scale production: Use HNSW with int8 quantization to balance query speed and retrieval quality.
  • Mind the indexing time: Factor in the significantly higher cost of building HNSW indexes if your data updates frequently.
  • Neural vs. Keyword: Use BM25 if extreme speed is required; otherwise, choose between dense and sparse models based on qualitative effectiveness for your specific domain.

Summary updated

Abstract

Practitioners working on dense retrieval today face a bewildering number of choices. Beyond selecting the embedding model, another consequential choice is the actual implementation of nearest-neighbor vector search. While best practices recommend HNSW indexes, flat vector indexes with brute-force search represent another viable option, particularly for smaller corpora and for rapid prototyping. In this paper, we provide experimental results on the BEIR dataset using the open-source Lucene search library that explicate the tradeoffs between HNSW and flat indexes (including quantized variants) from the perspectives of indexing time, query evaluation performance, and retrieval quality. With additional comparisons between dense and sparse retrievers, our results provide guidance for today's search practitioner in understanding the design space of dense and sparse retrievers. To our knowledge, we are the first to provide operational advice supported by empirical experiments in this regard.
BibTeX
@inproceedings{lin2025operational,
  author = {Jimmy Lin},
  title = {{Operational Advice for Dense and Sparse Retrievers: HNSW, Flat, or Inverted Indexes?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure