When a rumor spreads on social media, the replies give it away. False rumors draw more questions and denials than true ones, and systems that read the crowd's reaction can judge a claim they could not check directly. Scientific papers have a crowd too: the later papers that cite them. Each citing sentence (a "citance") restates the cited finding in another author's words, and it often signals agreement, qualification ("in a small murine cohort"), or contradiction ("we failed to replicate"). This proposal treats a paper's citances as its reply thread. We use them for two separate jobs that current scientific fact-checkers handle badly. First, citances bridge wording: they describe a finding in the vocabulary other people actually use, which helps retrieval when a claim and an abstract share few words. Second, they carry a stance signal: critical citances surface caveats and later disputes that the cited abstract never mentions.
This matters because open-domain scientific verification is failing in ways that more model capacity has not fixed. On SciFact-Open, a 500,000-abstract version of a biomedical claim benchmark, strong verifiers lose at least 15 F1 points. Mismatches in how specific a claim is compared with its evidence appear in 44% of evidence pairs, and about 20% of well-studied claims have conflicting evidence (SciFact-Open). Separately, misleading health claims tend to copy an abstract's conclusion, while the evidence that undermines them sits in methods or limitations text with little word overlap. When retrievers hand a language model only the matching "kernel of truth," Llama 3's rate of accepting false claims jumps from 0.0% to 44.4% (Grounding Fallacies). Citances are an evidence source written by people who have already read the paper critically, and current verifiers ignore it.
Core Claim
We claim that adding each candidate abstract's citances, with a stance label on each one, will improve open-domain scientific claim verification in two measurable ways. Indexing abstracts together with their citances should raise evidence recall on claims whose wording differs from the abstract. Letting the verifier weigh qualifying and contradicting citances should reduce false acceptance of overstated claims and improve accuracy on claims with conflicting evidence. The new ingredient is the citance evidence itself, meaning community stance aggregated over the citation graph. It is not a new verifier architecture. For that reason the design isolates how much of any gain comes from the citances and how much comes from model adaptation.
The closest prior work uses citances differently. Zero-shot Scientific Claim Verification Using LLMs and Citation Text uses a single citance as the claim to be checked against its cited abstract. It builds the SCitance set and shows that zero-shot GPT-4 reaches 80.1 micro-F1 there. It never gathers the many citances pointing at one paper, and it never uses them as evidence. Joint verifiers such as ARSJOINT (joint abstract retrieval, rationale selection and stance on SciFact) and VERT5ERINI (a BM25 plus T5 reranking-and-verification pipeline) judge each abstract in isolation. CoVer weighs conflicting evidence using crowd-written Community Notes, but only on social media claims. We borrow its idea of scoring evidence by directness and reliability and apply it to citing literature.
Research Program
The first component is a citance-augmented index. For every abstract in the S2ORC-derived SciFact-Open corpus, we collect the sentences in other S2ORC papers that cite it and index them as extra "views" of the abstract. The idea resembles MINDER (Li et al.), which indexes passages under synthetic pseudo-queries so user wording can reach the indexed text. Our views are real paraphrases written by domain experts, not generated ones, so they cannot introduce the invented premises that make HyDE-style expansion risky. A document's retrieval score is the maximum over its abstract view and its citance views. We run this with both BM25 and a dense retriever, because Comparing Knowledge Sources found BM25 more precise on expert claims and dense search stronger on lay phrasing. That finding constrains us: the index must help in both regimes rather than replace one of them.
The second component labels each citance as supporting, qualifying, contradicting, or neutral toward the cited finding, which mirrors the support, query, deny and comment categories of RumourEval. We avoid annotating citances by hand, which would be expensive. Instead we supervise the labeler weakly from claim-level labels, following the multiple-instance approach of A Weakly Supervised Propagation Model. In that approach, claim-level veracity alone trains post-level stance classifiers, because a refuted claim's thread must contain some denial. We add a small hand-checked set of about 500 citances only for evaluation.
The third component is a verifier that reads the claim, the abstract's rationale sentences, and up to $k$ citances ranked by relevance to the claim. We compare two versions. One is a frontier model used zero-shot. The other is a mid-size open model instruction-tuned on SciFact-style verification mixed with general instruction data. The mixing follows FLawN-T5, which found that continued pretraining on raw domain text gives nothing to large models and that domain-only instruction tuning damages general ability. The citance aggregation can also be written in closed form, as an interpretable reference point:
$$s(c,d) = s_{\text{abs}}(c,d) + \lambda \sum_{j \in C_d} w_j \, \sigma_j$$
Here $s(c,d)$ is the final support score for claim $c$ from document $d$, and $s_{\text{abs}}$ is the verifier's score from the abstract alone. $C_d$ is the set of retrieved citances of $d$, and $\sigma_j \in \{+1, 0, -1\}$ is citance $j$'s stance. $w_j$ is its relevance to the claim, and $\lambda$ is tuned on the SciFact development split. This is the averaged crowd posterior of CLEARumor, carried over from reply threads to citation graphs.
Baselines
The naive reference point is the SciFact-Open retrieval and verification pipeline run on abstracts only, with no citances. It isolates exactly what the new evidence adds.
The strongest published system on the target benchmark is MultiVerS, which reaches 52.4 F1 on SciFact-Open, down from 72.5 on the closed SciFact set (SciFact-Open). We rerun it on the identical corpus snapshot and claim split. Beating it means a statistically reliable gain in abstract-level F1 over 52.4, measured on SciFact-Open's pooled judgments together with our extra judgments described below.
A second strong competitor is the same frontier model given abstracts only. Few-shot GPT-4 already scores 72.2 F1 on closed SciFact (Zero-shot Scientific Claim Verification). We found no reported result on Lacuna for a frontier LLM on SciFact-Open, so we measure it ourselves.
A third baseline is VERT5ERINI-style BM25 plus T5 reranking. That reranking reportedly adds about 17 points in abstract retrieval, and the full pipeline scores 65.07 F1 against VERISCI's 55.31 on SciFact (VERT5ERINI).
Two ablations isolate the key ingredient. The citation-count ablation gives the verifier only how many times each paper was cited. The unlabeled-citance ablation gives it citances without stance labels. Every system receives the same retrieved candidate pool size, the same prompt-tuning budget of a fixed number of prompt variants chosen on the development split, and the same fine-tuning budget where fine-tuning applies.
Why not just…?
Why not just give the model the full text, including the methods and limitations sections? This is a real threat on MISCIPLUS, where the undermining context lives inside the single source paper. Grounding Fallacies shows that retrieval fails to reach that context, but it does not show that full-text reading fails. We therefore include full-text reading as a baseline. Citances can only win clearly on a second kind of claim: one where the source paper concedes nothing and later papers dispute it. Our conflict subset, described below, is built to test exactly that case. If full text matches citances on both, the stance component adds nothing, and we will report that.
Why not just expand the query with HyDE or train a better dense retriever? Hypothetical-document expansion helped on AVeriTeC, raising the answer-quality score from 0.187 to 0.2801 (HerO). However, it produces text that plausibly matches the claim, which tends to retrieve the supporting kernel rather than the caveat. We run HyDE and a dense retriever as retrieval baselines, and we also run them combined with citance views. The combination tests whether the two sources of wording overlap in what they recover or add to each other.
Why not just prompt a frontier model and skip fine-tuning? GPT-4's 80.1 on SCitance shows prompting can beat specialized models. Fine-tuning is therefore not the claim here. We report the zero-shot model with citances directly. If it matches the tuned open model, the contribution is the evidence source, and the tuned model's remaining value is lower cost and reproducibility. Those can be measured, but they are secondary.
Why not just use citation counts as the crowd signal? Popularity is not agreement. Heavily cited findings can be heavily disputed. The citation-count ablation measures directly whether stance labels add anything beyond counts.
Proposed Experiments
On SciFact-Open we report abstract-level retrieval recall at 10 and verification F1. One leakage hazard must be controlled. SciFact claims were originally written from citances, and SCitance maps them back. We therefore remove every citance drawn from the claim's source citing paper, and we report results with and without this filter. A second hazard is that SciFact-Open's gold evidence was pooled from four models, so a new retriever can find true evidence that was never judged. We have annotators judge the top five previously unjudged candidates for every system, using the original guidelines.
For wording mismatch that does not come from citances, we evaluate retrieval on the lay health claims of HEALTHFC and COVERT against PubMed. We found no reported per-dataset F1 on Lacuna for these, so we rerun BM25 and dense retrieval ourselves.
On MISCIPLUS we link its 100 publications to their S2ORC citances. We then measure two things: how often a qualifying citance points to the annotated undermining context, and the verifier's false acceptance rate against Llama 3's reported 44.4% when given only the kernel. We found no reported result on Lacuna for retrieving MISCIPLUS undermining passages, so the kernel-retrieval MRR of 0.664 serves only as context.
Finally, we form a conflict subset from the roughly 20% of SciFact-Open claims with opposing evidence. There we test whether citance stance improves accuracy over the unweighted majority vote used in Comparing Knowledge Sources.
The decisive check has three parts. On the filtered SciFact-Open split and on MISCIPLUS, the labeled-citance verifier must beat the abstract-only frontier model, the full-text baseline, and the unlabeled-citance ablation. If only the unlabeled version helps, citances are a wording aid and not a stance signal. If neither helps once leakage is filtered out, the idea is wrong.
References
- SciFact-Open: Towards open-domain scientific claim verification
- Grounding Fallacies Misrepresenting Scientific Publications in Evidence
- Zero-shot Scientific Claim Verification Using LLMs and Citation Text
- Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verification
- Scientific Claim Verification with VerT5erini
- Multiview Identifiers Enhanced Generative Retrieval
- Comparing Knowledge Sources for Open-Domain Scientific Claim Verification
- SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours
- A Weakly Supervised Propagation Model for Rumor Verification and Stance Detection with Multiple Instance Learning
- LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain
- CLEARumor at SemEval-2019 Task 7: ConvoLving ELMo Against Rumors
- HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims