When a question-answering or fact-checking system draws on several independent knowledge holders, such as hospitals, firms, or regional databases, it must decide which holders to trust. The best current methods decide by comparison. They collect the passages or answers each holder returns and reward holders that agree with the others. That approach fails in two settings that tend to occur together. First, holders often cannot hand over their passages, because those passages contain patient records or confidential business data. Second, a coordinated group of bad holders can form a majority and win any agreement contest. This proposal targets both failures with one mechanism.
A coordinating agent breaks each claim into small factual sub-questions, for example "which drug did cohort X receive?" or "what year did Y happen?". Each holder answers these privately and returns only a short typed answer, without the passages behind it. The coordinator mixes in hidden "probe" sub-questions whose answers it already knows from a public knowledge graph. Each holder's trustworthiness is then estimated from its accuracy on the probes, together with its agreement with other holders on the real sub-questions. The short answers make disclosure small and auditable. The probes supply ground truth that a colluding majority cannot outvote. The break-up into sub-questions is what allows holders with only partial coverage to be compared at all.
Core Claim
Credibility estimation across private sources does not require sharing passages. It needs comparable, low-content answers, plus a small supply of known-answer checks. We claim three things. First, sub-question answers carry enough signal to recover per-source reliability nearly as well as full-passage methods. Second, this signal can be obtained while disclosing much less private content than passage sharing or passage redaction. Third, known-answer probes drawn from a public graph keep the estimate correct when poisoned sources are a majority. That last condition is exactly where agreement-based methods are known to break.
Of the three claims, the third is the most distinctive. The first two are where a skeptic will push hardest.
Relation to Prior Work
The closest work is SecureCollaRAG. It builds a similarity graph over documents from many providers and uses a graph neural network to isolate poisoned clusters. On the MediNote clinical notes data it cuts the poisoning attack success rate to 5.24%, against 78.80% for plain multi-source retrieval. We differ from it in two respects. It requires every provider to export raw documents into the central graph, and it assumes malicious documents are a non-coordinating minority. We share no documents, and we explicitly test attacker majorities.
RA-RAG supplies part of our method. It estimates source reliability without labels by cross-checking answers over a shared query set, then combines answers by weighted majority vote. We reuse that estimator but change what it is applied to. RA-RAG cross-checks whole-question answers from sources that cover the same questions. We cross-check sub-question answers, which lets sources with partial or disjoint coverage be compared. We also add anchored probes, because unanchored agreement estimation falls into the majority-vote trap that both RA-RAG and the CONFACT study describe.
Knowledge-based trust scoring motivates the probes. That work scores a source by its overlap with verified facts rather than by link popularity. We apply this idea covertly and at the level of individual sub-questions.
The decomposition side builds on AtomR. AtomR splits a question into search, relate, and filter operators routed across knowledge graphs, text, and web search, and it reports a 9.5% F1 gain on BlendQA. It assumes that every source is readable and true. It also reports that 8B-parameter models cannot follow its planning syntax. This limitation constrains our design. Planning happens only in the coordinator, which sees nothing private. Holder agents only answer single sub-questions, so they can be small local models.
On the disclosure side, three papers shape the holder agent. LLMs Can't Play Hangman shows that chat agents without a private working memory cannot keep a secret and stay consistent. Our holder agent therefore keeps retrieved passages in an unexposed buffer. 1-2-3 Check shows that a separate checker role reduces leakage, with a 36% reduction on ConfAIde Tier 4 relative to a single agent. We reuse such a checker, but it screens a typed answer rather than a summary. DOSSIER is the closest precedent for "query locally, return only results": it achieved 75.1% on MIMIC-III claims using Text-to-SQL. It is limited to tables and trusts its sources. We extend the local-execution idea to free-text holders and to holders that may be adversarial.
Two negative results constrain the design. Stolba et al. show that naive noise injection destroys the coordination that multi-agent systems depend on. Can LLMs Keep a Secret? shows that chain-of-thought increases leakage. For both reasons, holder reasoning never leaves the holder.
Research Program
The system has one public coordinator and one agent per holder.
For each claim or question, the coordinator writes a tree of sub-questions whose answers are typed, such as an entity, a number, a date, or a yes/no stance. It sends each sub-question to every holder whose declared schema or topic could cover it. It also mixes in probe sub-questions of the same type and phrasing style. The probe answers come from Wikidata or SemMedDB, and the probes are templated to look like real traffic.
Each holder agent retrieves from its private store into private memory and answers in the typed slot or abstains. A local checker then rejects any answer that matches a disclosure policy. The policy covers identifiers, free text beyond the slot, and values flagged sensitive.
Reliability is estimated with a one-parameter-per-source model in the style of Dawid and Skene. On probe sub-questions the true answer $y_a$ is known, so the probe log-likelihood term for source $i$ is fixed. On real sub-questions the true answer is inferred by expectation-maximization. Aggregation uses the resulting log-odds weights:
$$\hat{y}_a=\arg\max_{y}\sum_{i\in S_a}\log\frac{v_i}{1-v_i}\,\mathbb{1}[y_{i,a}=y].$$
Here $S_a$ is the set of holders that answered sub-question $a$, $y_{i,a}$ is holder $i$'s answer, and $v_i$ is holder $i$'s estimated accuracy. This is the standard weighted-vote rule when errors are independent, which is also the rule RA-RAG uses. Fixing part of the likelihood to probe outcomes is what prevents a colluding majority from simply redefining the truth.
The research program has three analyses. The first measures how many probes are needed as the attacker fraction grows. We expect the error in $v_i$ to shrink roughly with one over the square root of the number of probes each source answers, since each probe is a Bernoulli sample of that source's accuracy. The second measures how much private content leaks per claim under an explicit extraction attack. The third measures when the typed-answer bottleneck loses accuracy relative to full-passage adjudication.
Baselines
The naive reference point is multi-source retrieval that pools all holders' passages into one generator prompt, with no reliability estimate. This is the setting SecureCollaRAG reports at 78.80% attack success rate on MediNote. It fixes the floor for robustness and the ceiling for disclosure.
The strongest robustness competitor is SecureCollaRAG run with full document access. Its reported 5.24% attack success rate against PoisonedRAG on MediNote with DeepSeek-V3 is the number to approach. We will re-run it on our exact partitions, using the same generator and the same retrieval budget. Our target is to come within a few points of its attack success rate while sharing no passages. Under attacker majorities, where it assumes the attack away, our target is to beat it outright.
The strongest agreement-based competitor is RA-RAG, run in its privacy-friendliest form. In that form holders return only whole-question answers and are cross-checked by weighted voting, with no probes. It gets the same holders, the same local models, and the same token budget. RA-RAG reports no accuracy number that we found on Lacuna for NQ, HotpotQA, or TriviaQA. It does report a 99.1% token reduction from source selection on TriviaQA. We will therefore report its accuracy from our own runs.
For heterogeneous clinical claims, DOSSIER serves as the local-execution baseline, at 75.1% accuracy with Claude-2 on MIMIC-III plus SemMedDB. For multi-hop claims, WKGFC provides the reference, at 72.8% balanced accuracy on HOVER. WKGFC is a non-private upper reference, not a like-for-like competitor.
Three ablations isolate the key ingredients: removing the probes, which reduces the method to decomposed RA-RAG; removing the decomposition, so probes are asked at whole-question level; and removing the local checker. All systems receive equal tuning budgets on a shared validation split.
Why not just…?
Why not just run RA-RAG on whole-question answers, which already avoids sharing passages? This is the main threat, and it may match us when holders cover the same questions and attackers are a minority. It has two weaknesses we expect to show. First, it cannot cross-check holders with disjoint coverage of a multi-hop claim, because they never answer the same question. Second, without anchors it is outvoted by a coordinated majority. The no-probe ablation and the attacker-fraction sweep settle this directly. If decomposed RA-RAG without probes equals the full system at every attacker fraction, then the probes are not the contribution, and we will say so.
Why not redact passages with a 1-2-3 Check-style checker and then run SecureCollaRAG? Redaction keeps the graph method's strength. However, DOSSIER and 1-2-3 Check both note that masking direct identifiers leaves the passages open to inference from correlated context. We will run redact-then-share as a baseline and attack it with the same extraction adversary. The adversary is a model that reads everything a holder disclosed and tries to recover planted canary attributes. If redaction reaches equal leakage at equal accuracy, the typed-answer bottleneck buys nothing.
Why not add differential-privacy noise to shared embeddings? Stolba et al. show that naive noise breaks coordination, and similarity-based clustering is exactly the kind of coupling that noise degrades. We include SecureCollaRAG on noised embeddings at several noise levels. This traces its accuracy–privacy curve against ours.
Why not let one frontier model read all silos with source background, as CONFACT recommends? CONFACT's source background augmentation gives a 10% absolute F1 gain for LLaMA-3.1 on its model-detected conflict split. It remains the right accuracy ceiling when disclosure is allowed. It is not an option when disclosure is forbidden, and Can LLMs Keep a Secret? reports 39% secret leakage for GPT-4 even with instructions. We report it as the non-private ceiling.
Proposed Experiments
We will build private holders from four sources. The first is MediNote, using Dirichlet non-IID partitions of PMC-Patients as in SecureCollaRAG, with planted canary attributes such as ages, identifiers, and rare diagnoses. The second is NQ and HotpotQA partitioned across holders, following RA-RAG's setup. The third is DOSSIER's MIMIC-III clinical claims, with SemMedDB as the public probe graph. The fourth is HOVER, with Wikipedia split across holders and Wikidata as the probe graph.
Poisoning uses PoisonedRAG and the Adaptive Tampering Attack. Collusion is swept from 10% to 70% of holders. The metrics are task accuracy or F1; attack success rate; rank correlation between estimated $v_i$ and each holder's true corruption rate; tokens disclosed per claim; and canary recovery rate under the extraction adversary.
The experiment we consider decisive is the attacker-majority sweep. There we expect both SecureCollaRAG and probe-free voting to collapse beyond 50% while the anchored estimator holds.
The second decisive check uses an adaptive attacker that also has access to the public graph. That attacker can answer every question found in the graph honestly and lie everywhere else. Against it we test two defenses. One draws probes from held-out verified facts that are not in the graph. The other perturbs templated facts. If neither defense keeps attack success rate low, the probe idea reduces to a defense against naive attackers, and we will report it as that.
References
- Combating Knowledge Corruption in Agent Systems: A Byzantine-Tolerant Secure Collaborative RAG Framework
- retrieval augmented generation with estimation of source reliability
- Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs
- Front Matter
- AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning
- LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents
- 1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning
- DOSSIER: Fact Checking in Electronic Health Records while Preserving Patient Privacy
- Privacy Leakage of Search-Based Multi-Agent Planning Algorithms
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
- Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checking