Making Literature RAG Follow the Evidence on Contested Alzheimer's Causal Links

Language models can now read hundreds of Alzheimer's disease (AD) papers and propose a causal map of the disease: which biomarker drives which, for example whether failing blood flow in the brain comes before amyloid plaque buildup. The best existing pipeline uses retrieval-augmented generation (RAG), meaning the model is shown passages pulled from a paper collection instead of answering from memory. It found something awkward. Larger models were less faithful to the retrieved passages than smaller ones. The likely cause is that the larger models had absorbed the amyloid-first story that dominated the literature of the 2000s, and that story overrode the text in front of them. That matters most for the links the field is actively revising. One example is the finding that vascular dysregulation precedes amyloid deposition. A literature-reading system that reverts to old dogma on exactly those links is least useful where it is most needed.

We propose to score each candidate causal link by how much the retrieved evidence moves the model, rather than by how confident the model is after reading it. For every passage, we compare the model's belief in a link with the passage against its belief with no passage. We record whether the passage supports one direction, the reverse direction, or neither. Then we aggregate these evidence-driven shifts across passages for both directions of the link. The system abstains when the evidence is balanced. Our claim is that this subtraction of the model's prior removes most of the scale-related faithfulness gap and resolves contested link directions more accurately. We test this against an expert causal graph and against a data-derived temporal ordering from the ADNI patient cohort.

Core Claim and Relation to Prior Work

The motivating work is Accelerating Causal Network Discovery of Alzheimer Disease Biomarkers via Scientific Literature-based Retrieval Augmented Generation. It builds a 200-paper open-access AD corpus. It retrieves snippets about biomarker pairs and asks LLMs with chain-of-thought prompting whether $A \rightarrow B$. It ranks links by output token probability and assembles them into an acyclic graph. It compares putting all snippets into one prompt ("concat-RAG") against judging each snippet separately ("split-RAG"). Its best result is roughly 0.66 link accuracy with Mixtral-8x7B and split-RAG. It also reports the faithfulness paradox: Llama-3.1-70B follows the retrieved text less than Mistral-7B or Mixtral.

We keep its corpus, task, and split-RAG idea of judging snippets one at a time. We change three things.

First, its confidence score is the model's probability after reading. That score mixes evidence with prior belief, which is exactly the channel through which the paradox operates. We subtract the no-evidence baseline instead.

Second, its per-snippet judgments are never asked about the reverse direction. Our system asks both directions explicitly. On contested pairs such as vascular dysregulation versus amyloid, the literature contains opposing claims, and recognizing that conflict is the point.

Third, it evaluates only against a clinician-curated graph. That graph may itself reflect the amyloid-first consensus. We add an independent, data-derived reference.

That reference comes from Early role of vascular dysregulation on late-onset Alzheimer's disease based on multifactorial data-driven analysis. Its authors reconstructed a disease timeline from 1,171 ADNI participants across five imaging modalities plus blood and spinal-fluid proteins. They place vascular dysregulation first, then amyloid deposition, then metabolic decline, then loss of functional connectivity, then structural atrophy. We use this ordering only as a consistency check on link direction for biomarker categories it covers. It is correlational, as the research file's open-problems section notes, so we do not treat it as causal ground truth. Its value is that its biases differ from those of an expert graph.

The general case for retrieval over memory in medicine is supported by results that motivate our setup without competing with it. Guideline-grounded RAG beat fine-tuning for hepatitis C management in From Guidelines to Real‐Time Conversation. A review of chronic disease LLM studies found grounding nearly tripled the odds of accurate answers in Unveiling the Potential of Large Language Models in Transforming Chronic Disease Management.

Those studies also show a tension that constrains our design. More context helped for hepatitis C guidelines, but more context hurt for AD link extraction, which is why split-RAG exists. We therefore keep evidence per-snippet. We also do not adopt graph-community retrieval as in KARE, which was built for ICU prediction. Our question concerns how retrieved evidence is weighed, not which neighborhood is retrieved.

Research Program

Let $p(\text{yes} \mid A \rightarrow B, d)$ be the model's probability of answering "yes, $A$ causes $B$" given snippet $d$. Let $\varnothing$ denote a neutral placeholder with no evidence. The evidence shift of snippet $d$ for that direction is

$$\Delta(A \rightarrow B, d) = \log p(\text{yes} \mid A \rightarrow B, d) - \log p(\text{yes} \mid A \rightarrow B, \varnothing).$$

This shift is near zero when the model would have said the same thing anyway. It is positive or negative only when the passage changes the model's mind.

For each snippet we also ask the same model one question with three options: does the passage state $A \rightarrow B$, $B \rightarrow A$, or neither? This yields stance probabilities $\pi_d$. The link score sums the shifts from snippets that address the link, weighted by how much each snippet is about that link:

$$S(A \rightarrow B) = \sum_{d} \pi_d(A \rightarrow B)\, \Delta(A \rightarrow B, d).$$

The system outputs the direction with the larger score when the margin $|S(A \rightarrow B) - S(B \rightarrow A)|$ exceeds a threshold tuned on held-out pairs. Otherwise it labels the pair "contested." Graph assembly then follows the original paper: rank links by score and run a cycle check.

We build this on the four open models the original study used (Llama-3.1-70B, Llama-3-8B, Mixtral-8x7B, Mistral-7B). We reuse their corpus and expert graph if the authors release them. If not, we rebuild a comparable 200-paper corpus spanning 2000–2025 under the same inclusion rules and have an AD clinician annotate a graph using their protocol. In that case we report that the numbers are not directly comparable to theirs.

Baselines

The do-nothing reference is each model answering every biomarker pair with no retrieved text. This measures pure parametric belief and is also the quantity our method subtracts.

Concat-RAG puts all top-$k$ snippets for a pair into one prompt. It represents the default RAG practice that the original study found diluted.

The strongest existing method is split-RAG with token-probability ranking from Accelerating Causal Network Discovery, whose best reported link accuracy is about 0.66 (Mixtral-8x7B). We rerun it rather than quote it. Every method uses the same corpus, the same retriever, the same top-$k$, the same prompt templates where they overlap, and the same per-pair inference budget, with our extra no-evidence call counted against that budget. Thresholds and $k$ for all methods are tuned on the same held-out pair split. Beating it means higher link accuracy and F1 on the expert graph for at least the two larger models. It also requires higher agreement with the ADNI-derived ordering on direction, without simply abstaining more often. We therefore report accuracy at matched coverage.

Two ablations isolate the key ingredient. The first keeps bidirectional querying and stance weighting but drops prior subtraction, scoring with raw post-evidence probability. The second keeps prior subtraction but asks only the forward direction. If the first ablation matches the full method, the subtraction is not doing the work.

Why not just…?

Why not just use a small model, since the paradox says small models are more faithful? The original study effectively did this, and Mixtral's roughly 0.66 is the number to beat. The issue is that small models are more faithful partly because they have weaker priors, not because they reason better about evidence. Abandoning the larger models gives up their reading ability. Our test is direct: if prior subtraction lets Llama-3.1-70B match or exceed Mixtral, the paradox is removable rather than a fixed property of scale. If small models still win after subtraction, we report that and the recommendation reduces to "use small models."

Why not just ask both directions and take the larger post-evidence probability? This is the most serious competitor because it is cheap and captures the bidirectional half of our idea. It is exactly our first ablation. We expect it to fail on contested pairs where the prior is strong in both readings. There, raw probabilities stay high for the dogmatic direction regardless of the snippet. If it matches the full method, we drop prior subtraction from the claim.

Why not just tell the large model to "answer only from the passage"? Instruction-following is the obvious fix, and the file contains no reported result for it on this task. We run it as an additional baseline on the large models with an identical budget. The counterfactual flip test below measures whether instructions actually change behavior or only change the wording of the answer.

Why not just count snippets that support each direction? Vote counting ignores how strongly a passage asserts a link and ignores the prior entirely. We include it, using the stance classifier alone with no shift term, as a lower-cost reference.

Proposed Experiments and Decisive Checks

The main evaluation reports link accuracy, F1, and direction accuracy on the expert graph. It also reports direction agreement with the ADNI-derived ordering on pairs drawn from vascular, amyloid, metabolic, connectivity, and atrophy markers. Results are broken out by whether a pair is "contested," meaning the corpus contains snippets asserting both directions. We expect our gains to concentrate there. Where the expert graph and the data ordering disagree, we report both rather than declaring a winner.

The decisive faithfulness check is counterfactual. For each snippet, we create a minimally edited version that reverses the stated direction, swapping cause and effect, and measure how often each method's decision flips accordingly. A faithful system should flip at nearly the rate that the edits warrant. We predict that baseline split-RAG on Llama-3.1-70B flips rarely on amyloid-related pairs, and that our method flips substantially more often there.

A second check uses the corpus's time structure. Restricting retrieval to pre-2010 papers and then to post-2015 papers should move our method's direction calls on the vascular–amyloid pair. A method dominated by its prior will barely change.

The proposal fails if the raw-probability ablation equals the full method. It also fails if gains appear only through increased abstention.