When Mouse Findings Become Human Facts: Measuring Qualifier Stripping in LLM Summaries of Neuroinflammation Research

Large language models (LLMs) are increasingly used to read and summarize biomedical literature. They make a specific, hard-to-see kind of factual error. The error is not inventing a gene or a drug. It is silently dropping the conditions under which a finding was observed. "Erdafitinib reduced LPS-induced NLRP3 activation in wild-type mice" becomes "erdafitinib suppresses neuroinflammation." Here LPS (lipopolysaccharide) is a bacterial toxin used to trigger inflammation artificially, and NLRP3 is an intracellular sensor protein that switches on inflammatory signaling. "Reactive astrocytes killed motor neurons in iPSC co-culture" becomes "astrocytes kill motor neurons in ALS." Each shortened sentence still names the right entities and the right link between them. Tools that check only whether that link exists will therefore mark it as grounded. Yet the shortened claim is false as a statement about human patients.

Neuroinflammation research is where this error does the most damage. The field's central finding is that neurons in Alzheimer's, Parkinson's and ALS are damaged largely by their support cells, the immune-like microglia and the metabolic-support astrocytes. The field's central problem is that this damage looks different in different experimental systems. Rodent glia respond differently from human glia. Transgenic mice develop plaques without the neuronal loss seen in patients. The glial feedback loop only appears in 3D human co-cultures, not in flat 2D cultures (Elucidating the Interactive Roles of Glia in Alzheimer's Disease). In this literature, the cell type, species, model system and triggering stimulus are part of what makes a claim true. We propose to measure how often LLMs strip these qualifiers. We also test whether a verifier that checks each qualifier separately can catch the errors that entity-level verifiers miss.

Core Claim

We hypothesize that most harmful LLM errors in preclinical neuroinflammation synthesis are qualifier stripping. By this we mean losing or upgrading the model system, cell type, stimulus, direction of effect, or hedging of a finding. We further hypothesize that entity-level verifiers score these stripped claims as grounded at nearly the same rate as faithful ones.

The closest existing work is Toward Reliable Biomedical Hypothesis Generation. It builds TruthHypo from PubTator 3.0 and verifies generated hypotheses with KnowHD, a method that splits text into atomic claims and checks each against knowledge graphs and PubMed. Its unit of truth is an entity pair: chemical–gene, disease–gene, or gene–gene. The paper itself reports a "how vs. that" gap, meaning models know that two entities interact but not how. We go one step further. We ask under which experimental conditions the interaction holds, and we make each condition a separately checked slot.

This is the step from coarse biomedical factuality to the kind of factuality neuroinflammation research actually needs. It also extends the "certainty illusion" found by Evaluating large language models on medical evidence summarization, where tentative findings become definitive statements. That work studied certainty illusions in clinical trial summaries. We study them in the preclinical-to-human translation step, where they are hardest to catch.

Research Program

The first component is a benchmark we call GliaQual. We will collect PubMed abstracts on glial neuroinflammation, using PubTator 3.0 entity annotations to seed candidates. Each abstract's key finding is annotated as a tuple with seven slots:

  • the intervening entity;
  • the target, which is a pathway or phenotype;
  • the direction of effect;
  • the cell type (microglia, astrocyte, neuron, or co-culture);
  • the model system, placed on an ordinal evidence tier (cell line, primary rodent culture, human iPSC 2D, human iPSC 3D co-culture, rodent in vivo, human tissue or cohort);
  • the inducing stimulus (for example LPS, amyloid-beta, a high-fat diet, or ischemia);
  • the hedge level.

The papers in our file show the typical shape of these findings. The 3-DSC study reports effects in LPS-stimulated microglia plus a traumatic brain injury model (3-Deoxysappanchalcone attenuates LPS-induced neuroinflammation). The erdafitinib study reports effects in wild-type mice given LPS (Erdafitinib diminishes LPS-mediated neuroinflammatory responses). The fasting study reports effects in diabetic mice through a gut microbe (Alternate day fasting alleviates neuroinflammation). Each is exactly the kind of finding that is easy to overgeneralize.

We will use a temporal split, following TruthHypo, so that the evaluated abstracts appear after the models' training cutoffs. Neuroscientists will annotate slots for a core set of findings. An LLM annotator will extend coverage, and its agreement with the human annotators will be reported.

The second component is a cross-condition diagnostic adapted from the Hallucination Dependence Index (The Hallucination Dependence Index). That paper compared ungrounded and evidence-grounded conditions on lung-cancer cases. It found that grounded accuracy compresses into a narrow, uninformative band while ungrounded behavior differs sharply across models. We use three conditions:

  • Closed-book: "What is known about X's effect on microglial activation?"
  • Single-abstract summarization.
  • Multi-abstract synthesis: five to ten abstracts about one pathway, mixing evidence tiers.

The third condition directly tests the "more is less" effect, in which more context produced more errors in Cochrane summarization (Evaluating large language models on medical evidence summarization). Our headline quantity is the tier upgrade rate:

$$U = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left[\tau(\hat{c}_i) > \tau(s_i)\right]$$

Here $N$ is the number of generated claims, $\hat{c}_i$ is the $i$-th claim, $s_i$ is the source finding it came from, and $\tau(\cdot)$ is the evidence tier either stated or implied by the text. A claim with no stated system is read as a claim about humans. We also report per-slot retention rates for the other qualifiers, and the rate at which the direction of effect flips.

The third component is a slot-wise verifier, which we call QualVerify. It breaks each generated claim into the same tuple. It then aligns each slot with the source abstract or abstracts and flags any slot that is missing, upgraded or contradicted. Its scores can be used to rerank Best-of-$N$ outputs, as KnowHD was used. The design choice that matters is that an absent qualifier counts as a failure, not as neutral. Entity-level groundedness and ordinary entailment checking both treat dropping a qualifier as harmless generalization.

The models under study are Llama-3.1-8B-Instruct, GPT-4o-mini and GPT-4o. The first is included because TruthHypo reports that smaller open models get worse when given retrieved context, so it probes the integration problem.

Baselines

The naive reference point is raw generation with no verification. We report $U$ and the slot retention rates for each model and condition. Every mitigation is measured against this.

The strongest existing competitor is KnowHD with Best-of-$N$ reranking, from Toward Reliable Biomedical Hypothesis Generation. It is the published method designed for biomedical hallucination in a closely related setting, and it improved GPT-4o-mini on TruthHypo chemical–gene tasks from 60% to nearly 73% accuracy with Best-of-5 selection. No reported result exists on Lacuna for KnowHD on qualifier-level errors or on any neuroinflammation-specific benchmark. We will run it on the same candidate pool as QualVerify, with the same $N=5$ candidates, the same PubMed retrieval corpus restricted by the same temporal cutoff, and a matched number of LLM calls. QualVerify beats it if it lowers $U$ and raises slot retention by a margin whose bootstrap confidence interval excludes zero, without lowering entity-and-direction accuracy.

The second baseline is a generic claim-level entailment checker, using the same LLM as judge, that asks whether each generated claim is supported by the source. This is the most obvious cheap alternative, and it shows whether slot structure matters at all. No reported result was found on Lacuna for entailment-based verification on biomedical qualifier errors.

The third baseline is prompting. We use a system instruction that requires the model to state species, cell type and model system. We also test Reflection of Thoughts prompting, which simulates multi-expert deliberation and backtracking. That method gave the highest guideline consistency reported in Prompt engineering in consistency and reliability with the evidence-based guideline: 62.9% overall for gpt-4-Web. That figure comes from an osteoarthritis guideline task, not ours.

The fourth baseline, used only for the multi-abstract condition, is DRAG. This is multi-agent debate over retrieved text, which reached 28.80 exact match on 2WikiMultihopQA at roughly 10 calls per question (Removal of Hallucination on Hallucination). We will compare it at matched call budgets.

The key ablation is QualVerify with only the entity and direction slots kept. This isolates whether the gains come from checking the qualifier slots specifically.

Why not just…?

Why not just scale to a frontier model? Scaling does not reliably buy truthfulness. TruthfulQA found larger models reproduced popular falsehoods more, with the best model at 58% against about 94% for humans. The HDI paper also saw a newer Gemini model regress on ungrounded confabulation. More directly, "LPS in mice suggests benefit in Alzheimer's" is a common way the literature itself talks. A larger model may therefore imitate this overgeneralization more fluently, not less. GPT-4o is included precisely to test this. If its upgrade rate is near zero, the scaling objection wins for frontier models, and the contribution narrows to smaller and open-weight models.

Why not just tell the model to state the species and model system? This is the most serious threat, and we treat it as a possible outcome rather than a strawman. For single-abstract summarization, a strong instruction may remove most stripping. If it does, we will report that and shift the verifier's role to the two conditions where prompting has no source to copy from. In the closed-book condition, the model must recall qualifiers it may never have stored. In the multi-abstract condition, qualifiers from different tiers must be kept attached to the right findings. Prompting's reported reliability on guidelines varied widely across configurations, with Fleiss' kappa from -0.002 to 0.984. That is a reason to measure it rather than assume it works.

Why not just use an entailment checker or KnowHD with a species-annotated knowledge graph? Entailment judges commonly accept a more general claim as supported by a more specific source. Entity-level graphs like PubTator record that a link exists, and at best the species, but not the stimulus, the dimensionality of the culture, or the hedge. Our first decisive check tests this directly: whether these checkers pass stripped claims at nearly the rate they pass faithful ones. If they do not, the gap we claim is not real, and we will say so.

Decisive Checks

Three results decide the proposal.

The first test is whether the gap exists at all. We create controlled perturbations of faithful claims by deleting or upgrading one slot at a time. We then check whether KnowHD and the entailment checker assign these perturbed claims groundedness scores statistically indistinguishable from the originals, while QualVerify separates them. We measure this separation with AUROC.

The second test checks whether the "more is less" effect appears as qualifier loss. We look for whether $U$ rises from the single-abstract to the multi-abstract condition, especially for Llama-3.1-8B. This would connect the context paradox to a specific mechanism: qualifiers getting detached from their findings.

The third test concerns QualVerify-guided Best-of-5 reranking. We ask whether it lowers $U$ and raises human-judged translational accuracy relative to KnowHD at matched compute, and whether the slot ablation loses most of that gain. We will also report the failure case honestly: findings where the source abstract itself overstates its model system. In those cases, faithfulness to the source and truth about humans come apart, and neither kind of verifier can resolve the conflict.