Researchers who cannot keep up with the volume of new papers increasingly ask large language models (LLMs) to summarize and synthesize findings for them. The usual worry is hallucination, meaning the model states things the sources do not say. A quieter failure deserves equal attention. The model can report a claim the source really does make while changing how sure the source was about it. "Our results suggest Drug A may reduce symptoms" becomes "Drug A reduces symptoms." Every word of the fact is supported, so a standard support check passes it. Yet the summary has turned a hypothesis into a finding. We call this certainty drift. We propose to measure it with tools from epistemic modality detection, the subfield that identifies the words and grammatical structures by which authors signal how strongly they commit to a claim.
The second half of the idea is about fairness in literature review. Authors writing in English as a second language, and papers translated from other languages, hedge differently: some hedge more, some less, and some use modal verbs in unusual ways. If LLM summarizers flatten or inflate certainty unevenly across these writing styles, then automated synthesis does more than add noise. It systematically promotes some research communities' claims and demotes others'. We propose a claim-level audit that compares the certainty of each claim in the source with its certainty in the LLM summary. We then ask whether drift differs between papers by native-English authors, by non-native authors, and papers originally written in other languages. Finally, we test whether simple fixes remove the drift evenly across these groups.
Core Claim
Hallucination checks ask whether a summarized claim is supported by its source. We argue they should also ask whether the claim's epistemic status is preserved, where epistemic status means confirmed, tentative, or negated. Answering that question requires claim-level rather than sentence-level modality analysis, because a single scientific sentence often mixes established background with speculation. We further hypothesize that certainty drift in LLM summaries is not uniform across author language background. If so, it adds a new and measurable route by which language bias enters automated literature review.
The novelty lies in how existing tools are pointed and paired, not in the detector itself. The closest existing work is the SpecTup line, which classifies Open Information Extraction tuples as factual or speculative. Open Information Extraction means pulling (subject, relation, object) triples from free text without a fixed schema. That work is reported in Shall We Trust All Relational Tuples by Open Information Extraction? A Study on Speculation Detection, From Speculation Detection to Trustworthy Relational Tuples in Information Extraction and Dong et al.. It labels claims inside human-written text so that extracted knowledge bases do not treat speculation as fact. We reuse that approach as a measuring instrument and apply it twice: once to the source and once to a machine-written summary of it. We then study the difference between the two readings and how that difference splits across author populations.
The research file I worked from was truncated after its overview of epistemic modality detection. It contained no section on LLM evaluation, literature overload or language bias. I found no prior work in it that audits LLM summaries for preserved certainty, but that is not proof that none exists.
Research Program
The program has three parts.
Part 1: the claim-level certainty auditor. For each source abstract or results paragraph, we extract claims as relational tuples and attach an epistemic label to each: confirmed, tentative, or negated. We may add a graded certainty score where annotation supports it.
The design follows three established results. First, uncertainty is carried by cue words whose speculative sense must be disambiguated, as Velldal shows. Second, the reach of a cue over the sentence follows syntax rather than word order, as Øvrelid et al. and Zhang et al. show. Third, attaching certainty to the claim itself rather than to the sentence keeps background facts and speculation apart. This third point is the core idea of the reified claim graphs in SciClaim and of the meta-knowledge annotations for biomedical events by Thompson et al. and Nawaz et al..
The auditor combines three components:
- a SpecTup-style encoder that pairs a BERT text encoder with a graph network over the dependency neighbors of the relation word;
- a modal-verb function classifier in the style of MiST, because "may" and "could" frequently signal possibility or permission rather than doubt;
- an LLM-based extractor, included as a cheaper alternative whose accuracy we measure rather than assume.
We run the same auditor on the LLM summary. We then align summary tuples to source tuples, using entity matching plus embedding similarity with human-checked thresholds. Certainty drift for an aligned pair is the signed change in epistemic level from source to summary. Upgrades, meaning tentative becoming confirmed or a negation being dropped, are tracked separately from downgrades.
Part 2: measuring drift by author population. We build three groups of papers matched on topic, venue tier and publication year:
- papers whose authors are all at institutions in English-speaking countries;
- papers whose authors are all at institutions in non-English-speaking countries;
- papers originally published in another language, represented by their English abstracts and by machine translations of their native-language abstracts.
Institution country is a crude proxy for language background. We report it as such and check the proxy on a hand-verified subsample.
We then ask several current LLMs to produce single-paper summaries and multi-paper syntheses. A multi-paper synthesis has the model write an answer to a review-style question from five to twenty abstracts. In that setting drift can also take the form of a tentative minority result disappearing behind a confident majority. We report three quantities per group: the drift rate, the direction of drift, and how often a group's claims are cited or omitted in the syntheses.
Part 3: fixes, and whether they help evenly. We compare three interventions:
- a prompt instructing the model to preserve the authors' level of certainty;
- a structured pipeline in which the model receives the source tuples with their epistemic tags and must carry the tags into its output;
- post-hoc repair, in which the auditor flags drifted claims and the model rewrites only those sentences.
The question that matters is whether each fix shrinks the gap between groups, not only the average drift.
Human ground truth. Ground truth for all three parts comes from expert annotation of several hundred source–summary claim pairs per group. Annotators judge whether certainty was preserved, upgraded, or downgraded. Inter-annotator agreement sets the ceiling against which the automated auditor is judged.
Baselines
Naive reference. The do-nothing baseline is a plain LLM summary with no certainty instruction. It is scored by two auditors: a conventional support check (is each summary claim supported by the source?) and our certainty auditor. If the support check rates these summaries as faithful while humans and our auditor find frequent upgrades, that gap is the core motivating result.
Strongest existing detector. The strongest existing method for labeling claim-level speculation is the SpecTup architecture, run on source text exactly as published and on summary text unchanged. I found no reported result on Lacuna for SpecTup, MiST or SciClaim in the file, so I cannot quote a published figure. We will retrain each from its released resources where available, under a matched tuning budget.
Sentence-level classifier. This baseline assigns one certainty label to an entire sentence, in the tradition of Riloff et al.. It is the natural cheaper competitor to our claim-level auditor.
Ablations. These isolate the key ingredient. We remove the syntactic graph component, leaving a text-only encoder, and separately remove the modal-function classifier. This shows whether syntax and modal disambiguation actually matter on LLM-written text. That is not guaranteed, because LLM prose may hedge in simpler, more regular forms than human prose.
Fair comparison. All auditors see the same aligned pairs, receive the same annotated training split, and get the same hyperparameter search budget.
What counts as beating a baseline. Beating the sentence-level classifier means higher agreement with human drift judgments on sentences that contain more than one claim. Beating SpecTup-as-is means higher macro-F1 on drift direction on summary text. On the intervention side, a fix beats the prompt baseline only if it lowers both mean drift and the between-group gap, with confidence intervals from bootstrapping over papers.
Why not just…?
Why not just prompt the model to preserve hedges? This is a real threat to the intervention part. Modern LLMs follow such instructions reasonably well, and it may cut average drift sharply. That is why the proposal does not rest on beating the prompt. The prompt is itself one of the measured conditions, and the question is whether its benefit is uniform across groups. An instruction to "keep the authors' certainty" depends on the model recognizing the hedge in the first place. Hedges expressed through unusual modal-verb use or translated phrasing are exactly where we expect recognition to fail, given the sense ambiguity documented by Velldal and MiST. If prompting closes both the average drift and the gap, we report that plainly, and the contribution becomes the audit and the measurement of the gap.
Why not just use an entailment-based factual consistency checker? Entailment checking asks whether the source logically implies the summary. In principle it should notice that "may reduce" does not entail "reduces," so it is the most serious competitor to our auditor. We will run a strong entailment checker on the same pairs. It may catch upgrades but miss downgrades, since a stronger source does entail a weaker summary. It also gives no claim-level location or direction. If its agreement with human drift labels matches our auditor's, we will adopt it and drop the custom detector.
Why not just classify certainty per sentence? It is simpler and needs less annotation. However, the SciClaim and biomedical-event literature motivates claim-level labels precisely because sentences mix confirmed and speculative content. The multi-claim sentence subset in our human annotation directly tests whether this matters for summaries.
Why not just translate everything to English first and treat it uniformly? Translation is part of the problem being studied, not a neutral preprocessing step. Comparing native-language abstracts that we machine-translate against the authors' own English abstracts shows whether translation itself adds or removes hedges before any summarizer sees the text.
Decisive Experiments
Experiment 1: is drift real? On human-annotated pairs, we measure how often plain LLM summaries upgrade tentative or negated claims, and whether conventional support checks flag those cases. A high upgrade rate that support checks miss establishes the phenomenon. A rate near annotator noise would end the project early and cheaply.
Experiment 2: is the auditor trustworthy on machine text? Our auditor must reach agreement with humans close to inter-annotator agreement, and it must beat both the sentence-level and entailment baselines on drift direction. Otherwise, downstream group comparisons rely on human labels alone at a smaller scale.
Experiment 3: does drift differ by group? We test whether drift rates and directions differ across the three author populations after controlling for topic, abstract length and base hedge density. The base-density control matters. A group that hedges more has more hedges to lose, so we report drift conditional on the source's epistemic label, not raw counts.
Experiment 4: do fixes help evenly? We test whether each intervention narrows the between-group gap or only the average. A fix that lowers average drift while widening the gap would be the most consequential finding. It would mean that standard hallucination mitigation can improve faithfulness benchmarks while deepening language bias in automated literature review.
References
- Shall We Trust All Relational Tuples by Open Information Extraction? A Study on Speculation Detection
- From Speculation Detection to Trustworthy Relational Tuples in Information Extraction
- Detecting uncertainty in biomedical literature: a simple disambiguation approach using sparse random indexing
- Syntactic Scope Resolution in Uncertainty Analysis
- Hedge Detection and Scope Finding by Sequence Labeling with Procedural Feature Selection
- Extracting Fine-Grained Knowledge Graphs of Scientific Claims: Dataset and Transformer-Based Results
- Evaluating a meta-knowledge annotation scheme for bio-events
- Meta-Knowledge Annotation at the Event Level: Comparison between Abstracts and Full Papers
- MiST: a Large-Scale Annotated Resource and Neural Models for Functions of Modal Verbs in English Scientific Text
- Exploiting Subjectivity Classification to Improve Information Extraction