ZeroRAG: Adaptive Semantic Peer Routing for Decentralized Retrieval-Augmented Generation

Petru Neague, Quinten Stokkink, Martijn de Vos, Rafael Pires, Mathis Randl, Anne-Marie Kermarrec, Johan PouwelsearXiv 2026 · DOI · arXiv

TL;DR

ZeroRAG is a decentralized retrieval layer for Retrieval-Augmented Generation (RAG) that eliminates the need for a central document store or a global index. By using a self-organizing peer-to-peer overlay that routes queries based on semantic similarity, it achieves near-centralized performance while keeping data firmly in the hands of its original owners.

What Problem Does This Paper Address?

Standard RAG pipelines rely on a centralized vector database to store and index document chunks (see Figure 1). While efficient, this architecture is a non-starter for many real-world scenarios. Organizations like hospitals or law firms often cannot surrender custody of their data due to privacy regulations (HIPAA, GDPR), copyright concerns, or simple data sovereignty.

Figure 1: The standard RAG pipeline: offline indexing (left) encodes documents into a vector database; online retrieval (right) matches a query to relevant chunks before LLM answer generation.

Existing decentralized retrieval methods have significant drawbacks: they either re-introduce a coordinating authority (defeating the purpose of decentralization) or rely on rigid global data structures like Distributed Hash Tables (DHTs) that require constant, network-wide maintenance. ZeroRAG aims to provide a "best-effort" retrieval layer that is fully decentralized, requires no global coordination, and scales to hundreds of thousands of peers.

Methods

ZeroRAG treats retrieval as a querier-based local routing problem. The system is built on three core pillars:

  1. Lightweight Local State: Instead of a global index, each peer maintains only a small list of neighbors, compact "advertisements" (embedding vectors) of what those neighbors hold, and a cache of recently routed queries.
  2. Best-First Semantic Walk: When a peer issues a query, it doesn't broadcast it to everyone. Instead, it performs a greedy walk through the network. It scores its neighbors based on their advertisements and its own query cache, then asks the most promising peers for results and referrals to their neighbors.
  3. Adaptive Topology: This is the "secret sauce." The network is not static. After a successful query, the querier adds a direct connection to the peer that provided the best result, evicting the oldest neighbor if necessary. Over time, this implicit feedback loop causes the overlay to self-organize, clustering peers with semantically similar data or interests.
Figure 2: (left) Example of 4 query steps of a document searched by peer 𝑎with query 𝑞. After the search concludes, peer 𝑎adds peer 𝑖, which has the most relevant document to 𝑎’s query, as neighbor. (right) The dynamics of query caching in ZeroRAG.

As shown in the figure above, ZeroRAG also uses query caching. If Peer B recently helped Peer A find a document about "ACE inhibitors," and Peer H later asks a similar question, Peer B can refer Peer H directly to Peer A.

Datasets

The authors evaluated ZeroRAG using three diverse real-world workloads:

  • AOL-IA: A query-and-click stream used to test single-target retrieval (finding one specific document).
  • Amazon Electronics: A product-search benchmark used to test multi-relevant retrieval (finding many related products).
  • PubMedQA: A biomedical question-answering dataset used for end-to-end RAG evaluation, where retrieved documents are fed into a Phi-3.5-mini-instruct model to generate answers.

Key Insights and Ideas

  • Routing by Interest, not just Content: ZeroRAG realizes that a peer's query history is as good a routing signal as its local documents. If you've asked about a topic recently, you likely know who has the answers.
  • Emergent Semantic Maps: The network doesn't need a map; it becomes the map. Through continuous rewiring, the topology naturally aligns with the semantic distribution of data across the network.
  • Progressive Retrieval: The system can act as a "fast-or-deep" mechanism. A peer can stop after a few contacts for a quick, approximate result or continue the walk to maximize recall at the cost of more messages.
  • Self-Healing Resilience: Because routing state is rebuilt from incoming traffic, the network can recover from catastrophic failures (like 50% of peers leaving) without any manual intervention or global re-indexing.

Key Results

  • Superior Recall: On the AOL-IA workload, ZeroRAG achieved a recall of 0.91, significantly outperforming decentralized baselines like Semantica (0.65) and DHT-LSH (0.80).
  • Near-Centralized Accuracy: In end-to-end QA tasks, ZeroRAG reached 77% accuracy, trailing a centralized HNSW index (79%) by only a small margin.
  • Extreme Efficiency: As the overlay organizes, the number of messages drops. In its final state, 99% of queries found the best document in 13 or fewer peer contacts.
  • Scalability: The system maintained high recall (above 98%) while scaling from 1,000 to 100,000 peers.
  • Robustness: After a simulated "catastrophic failure" where half the network crashed, ZeroRAG self-healed back to 95.5% recall through local interactions alone.

Limitations and Future Work

  • Best-Effort Guarantees: Unlike a DHT, ZeroRAG is probabilistic. It cannot strictly guarantee that a document will be found even if it exists in the network, though the empirical recall is very high.
  • Privacy Leaks: While documents never leave the owner's device, the "advertisements" and query referrals could potentially leak information about what a peer is interested in or what data it holds.
  • Byzantine Faults: The current model assumes peers are honest. The authors note that investigating malicious behavior (e.g., peers trying to bias search results or "black hole" attacks) is a direction for future work.
  • Cold Start: The system performs best after it has had time to self-organize. Initial queries in a brand-new network will have lower recall and higher latency.

Summary updated

Abstract

Retrieval-augmented generation (RAG) is a standard approach to enhance the reliability of large language model (LLM) outputs by retrieving documents relevant to a query and supplying them in the LLM prompt. Today's RAG systems collect, index, and serve these documents from centralized stores. In practice, however, documents are owned by organizations, devices, and individuals who cannot or will not surrender custody of them to a central party, e.g., because of privacy, copyright or regulatory compliance. Existing modern decentralized retrieval techniques either reintroduce a coordinating authority, or rely on a global data structure that requires continuous network-wide coordination (e.g., DHTs). We present ZeroRAG, a decentralized routing layer for document retrieval in RAG that performs semantic query routing across an overlay of peers with- out global data structures or centralized orchestration. Documents remain with the peers that own them, and each peer keeps only lightweight local state: a bounded set of neighbor advertisements and a cache of recently routed queries. Using these, ZeroRAG steers each query through a best-first walk that progressively rewires the overlay toward semantically relevant neighbors, so recall improves over time as the network self-organizes. We evaluate ZeroRAG using three real-world workloads and two decentralized baselines. We find that ZeroRAG outperforms baselines in terms of recall and required network messages, approaches the recall and LLM answer accuracy of a centralized document store, self-heals after catastrophic peer failures, and scales to 100 000 peers.
BibTeX
@misc{neague2026zerorag,
  author = {Petru Neague and Quinten Stokkink and Martijn de Vos and Rafael Pires and Mathis Randl and Anne-Marie Kermarrec and Johan Pouwelse},
  title = {{ZeroRAG: Adaptive Semantic Peer Routing for Decentralized Retrieval-Augmented Generation}},
  year = {2026},
  eprint = {2611.07928},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07928},
}
Figure