TL;DR
ZeroRAG is a decentralized retrieval layer for Retrieval-Augmented Generation (RAG) that eliminates the need for a central document store or a global index. By using a self-organizing peer-to-peer overlay that routes queries based on semantic similarity, it achieves near-centralized performance while keeping data firmly in the hands of its original owners.
What Problem Does This Paper Address?
Standard RAG pipelines rely on a centralized vector database to store and index document chunks (see Figure 1). While efficient, this architecture is a non-starter for many real-world scenarios. Organizations like hospitals or law firms often cannot surrender custody of their data due to privacy regulations (HIPAA, GDPR), copyright concerns, or simple data sovereignty.
Existing decentralized retrieval methods have significant drawbacks: they either re-introduce a coordinating authority (defeating the purpose of decentralization) or rely on rigid global data structures like Distributed Hash Tables (DHTs) that require constant, network-wide maintenance. ZeroRAG aims to provide a "best-effort" retrieval layer that is fully decentralized, requires no global coordination, and scales to hundreds of thousands of peers.
Methods
ZeroRAG treats retrieval as a querier-based local routing problem. The system is built on three core pillars:
- Lightweight Local State: Instead of a global index, each peer maintains only a small list of neighbors, compact "advertisements" (embedding vectors) of what those neighbors hold, and a cache of recently routed queries.
- Best-First Semantic Walk: When a peer issues a query, it doesn't broadcast it to everyone. Instead, it performs a greedy walk through the network. It scores its neighbors based on their advertisements and its own query cache, then asks the most promising peers for results and referrals to their neighbors.
- Adaptive Topology: This is the "secret sauce." The network is not static. After a successful query, the querier adds a direct connection to the peer that provided the best result, evicting the oldest neighbor if necessary. Over time, this implicit feedback loop causes the overlay to self-organize, clustering peers with semantically similar data or interests.
As shown in the figure above, ZeroRAG also uses query caching. If Peer B recently helped Peer A find a document about "ACE inhibitors," and Peer H later asks a similar question, Peer B can refer Peer H directly to Peer A.
Datasets
The authors evaluated ZeroRAG using three diverse real-world workloads:
- AOL-IA: A query-and-click stream used to test single-target retrieval (finding one specific document).
- Amazon Electronics: A product-search benchmark used to test multi-relevant retrieval (finding many related products).
- PubMedQA: A biomedical question-answering dataset used for end-to-end RAG evaluation, where retrieved documents are fed into a Phi-3.5-mini-instruct model to generate answers.
Key Insights and Ideas
- Routing by Interest, not just Content: ZeroRAG realizes that a peer's query history is as good a routing signal as its local documents. If you've asked about a topic recently, you likely know who has the answers.
- Emergent Semantic Maps: The network doesn't need a map; it becomes the map. Through continuous rewiring, the topology naturally aligns with the semantic distribution of data across the network.
- Progressive Retrieval: The system can act as a "fast-or-deep" mechanism. A peer can stop after a few contacts for a quick, approximate result or continue the walk to maximize recall at the cost of more messages.
- Self-Healing Resilience: Because routing state is rebuilt from incoming traffic, the network can recover from catastrophic failures (like 50% of peers leaving) without any manual intervention or global re-indexing.
Key Results
- Superior Recall: On the AOL-IA workload, ZeroRAG achieved a recall of 0.91, significantly outperforming decentralized baselines like Semantica (0.65) and DHT-LSH (0.80).
- Near-Centralized Accuracy: In end-to-end QA tasks, ZeroRAG reached 77% accuracy, trailing a centralized HNSW index (79%) by only a small margin.
- Extreme Efficiency: As the overlay organizes, the number of messages drops. In its final state, 99% of queries found the best document in 13 or fewer peer contacts.
- Scalability: The system maintained high recall (above 98%) while scaling from 1,000 to 100,000 peers.
- Robustness: After a simulated "catastrophic failure" where half the network crashed, ZeroRAG self-healed back to 95.5% recall through local interactions alone.
Limitations and Future Work
- Best-Effort Guarantees: Unlike a DHT, ZeroRAG is probabilistic. It cannot strictly guarantee that a document will be found even if it exists in the network, though the empirical recall is very high.
- Privacy Leaks: While documents never leave the owner's device, the "advertisements" and query referrals could potentially leak information about what a peer is interested in or what data it holds.
- Byzantine Faults: The current model assumes peers are honest. The authors note that investigating malicious behavior (e.g., peers trying to bias search results or "black hole" attacks) is a direction for future work.
- Cold Start: The system performs best after it has had time to self-organize. Initial queries in a brand-new network will have lower recall and higher latency.
Summary updated