Is the Criticism Earned? Checking Evaluative Citations Against the Cited Paper

When one paper cites another, it often passes judgment: "X fails on long documents," "Y's evaluation is limited to English," "Z substantially improves over prior work." Researchers, reviewers and citation-based impact metrics all read these statements as evidence about the cited work. Yet no automated system checks whether they are true. Citation sentiment analysis, the task of labeling a citation as positive, negative or neutral from the surrounding text, only asks what the citing author said. Citation verification tools only ask whether the reference exists and is on topic. We propose a third question: given the full text of the cited paper, is the citing author's praise or criticism supported, contradicted, or impossible to check from that paper alone?

The natural tool for this is an LLM-as-a-judge, meaning a large language model prompted to reach a verdict about a piece of text. Recent evidence, however, suggests that such judges are swayed by exactly the features scientific citations are full of. They reward fluent, confident analytical prose over real evidential reasoning, and they defer to famous papers. Our central claim is that a judge that reads the citing author's rhetoric will let hedging, politeness and confidence leak into its verdict about fidelity. Separating the steps fixes this. First, rewrite each evaluative citation as a plain factual assertion about the cited paper, with the rhetoric removed. Then judge that assertion against retrieved passages from the cited paper, with the cited paper's identity hidden. We will build a benchmark of human-annotated evaluative citations to test this claim, together with controlled rewrites that change only the rhetoric or only the facts. The benchmark measures accuracy, and it also measures whether judges keep their verdict when only the style changes and change it when the facts change.

What the two directions contain, and where they fail to meet

Citation sentiment work has spent a decade learning to detect evaluation. Context-Enhanced Citation Sentiment Detection showed that reading the sentences around a citation more than doubles the share of citations found to carry positive or negative sentiment, and triples the negative share. The implication is that criticism is usually softened and spread out. MultiCite established that 17.1% of citation contexts span several sentences and that the reason for a citation is often not expressed in the sentence containing it. The strongest current classifier, TDCSA, has LLaMA-3.1 models extract structured "aspect, opinion, polarity" tuples and check them with chain-of-thought reasoning, then feeds these into a fine-tuned RoBERTa. It reaches 81.7 Macro-F1 on the Athar corpus, against 68.0 for plain RoBERTa. These papers motivate our work, because they show that evaluative citations are frequent, subtle and recoverable. They also supply our first stage, since we need to find the evaluative citations and their extent before judging them. None of them opens the cited paper.

Work on LLM judges supplies the constraints. Articulate Intuition or Genuine Analysis? had Claude-3-7-sonnet rate 3,563 peer reviews. The judge collapsed nine rubric dimensions into a single "sounds analytical" factor, and review length explained about 74% of the score advantage of AI-written reviews. That judge never saw the manuscripts under review, so it could only reward style. Curse of Knowledge showed that giving judges reference material and rubrics can make them less robust, and that reasoning models degraded more than twice as much as general models on biased samples (3.35% vs. 1.27% for DeepSeek-R1 vs. DeepSeek-v3). Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias found that LLMs favor highly cited papers: their suggested references had a median of 1,326 more citations than the true ones. We take these results to mean that a judge reading "the widely used method of X unfortunately struggles with…" next to X's paper may be moved by the politeness, the confidence and X's fame. Our design is built to remove each of these channels.

The closest system, CiteGuard, combines keyword and SPECTER2 embedding retrieval, an isotonic-calibrated alignment score and a 4B LLM that handles the uncertain 18% of cases. It reaches 0.89 citation-level F1 on REALCITATIONERRORS-500. But its error classes are bibliographic: dead DOIs, retractions, metadata mismatch and topical drift. A citation saying "X fails on long documents" is highly on-topic for X's abstract whether or not it is true, so this kind of check cannot flag misrepresentation. Explaining Relationships Among Research Papers generates descriptions of how papers relate but does not assess whether stated relations are faithful. InnoEval uses retrieval-grounded panels of persona judges for research ideas, a method we borrow only as a stronger baseline. It never judges individual citations.

Core Claim

Judging whether an evaluative citation is faithful to the cited paper is a separate task from classifying its sentiment. Judges that see the citing author's original wording will show measurable sensitivity to rhetoric, meaning their verdicts change when only hedging or politeness changes. They will also show sensitivity to the cited paper's popularity. A pipeline that strips out rhetoric, hides the cited paper's identity and grounds the judgment in retrieved evidence will cut both sensitivities substantially without losing accuracy. If a vanilla frontier judge that reads the full cited paper is already invariant to rhetoric and popularity, the decomposition is unnecessary. The benchmark and the invariance tests would then be the contribution, and we would say so.

Research Program

The first component is a benchmark we call CiteFidelity. We draw citation contexts from MultiCite and the Athar corpus, both from the ACL Anthology. We restrict them to cited papers whose full text is also in the Anthology, so evidence is available. We keep contexts labeled positive or negative, and those labeled comparison or motivation in MultiCite. Two annotators with NLP research experience read each context and the cited paper. They first rewrite the evaluation as one or more atomic assertions, then label each assertion with one of four verdicts. The assertion is supported by the cited paper, contradicted by it, not addressed by it, or not checkable from it. The last category covers assertions that depend on the citing paper's own experiments or on pure opinion. That label matters, because a large share of criticism ("X performed poorly on our data") is legitimately outside the cited paper's scope. A judge that calls such assertions "unsupported" is making a category error. We target around 1,500 assertions with full double annotation and report Cohen's kappa, which sets the ceiling any judge can be compared against.

Natural misrepresentations may be rare, so we add two kinds of controlled rewrites, written by annotators and validated by a second annotator. Rhetorical rewrites keep the content and change the style: a hedged critique made blunt, a praise-then-criticism sentence reduced to the criticism, a confident claim softened. The verdict should not move. Factual rewrites keep the style and change one specific detail, such as the dataset, the metric, the direction of a comparison or the size of an effect, so that a supported assertion becomes contradicted. The verdict should move. Together these give two behavioral measures beyond accuracy. The rhetorical flip rate is the fraction of rhetorical rewrite pairs on which the judge's verdict differs. Evidence sensitivity is the fraction of factual rewrite pairs on which the verdict changes in the correct direction. For popularity, each item is judged twice, once with the cited paper's title and authors visible and once anonymized. We report the verdict difference by cited-paper citation count.

The second component is the decomposed judge, which has three stages. A span finder picks the citation's context boundaries using MultiCite's gold boundaries as training signal. MultiCite reports 85.61 Macro-F1 for context identification with a four-sentence window. Choosing only the relevant span, rather than a fixed window, addresses the "context paradox" in Scientific Software Citation Intent Classification Using Large Language Models, where adding sentences hurt. An assertion extractor, a prompted LLM checked against the annotators' rewrites, then turns the span into neutral atomic assertions with polarity words, hedges and the cited paper's name removed. It also tags each assertion as within the cited paper's scope or outside it. Finally, a verifier sees only one neutral assertion and the top retrieved passages from the anonymized cited paper, and returns a verdict with the quoted evidence. A verdict without a quoted span is not allowed to be "contradicted," which forces the verifier to point at text rather than write an analytical-sounding argument.

Baselines

The naive reference is a majority-class predictor, almost certainly "supported." Next to it sits a polarity-only heuristic that maps TDCSA's predicted polarity to a verdict, treating positive as supported and negative as contradicted. That heuristic shows how much of the task is just sentiment, and it is the baseline to beat on the contradicted class. TDCSA's reported 81.7 Macro-F1 on Athar is for polarity, not fidelity. There is no reported result on Lacuna for the evaluative-citation fidelity task itself, so every baseline number on CiteFidelity will be new.

The retrieval-only competitor reproduces CiteGuard's scoring stack: SPECTER2 similarity between the citation context and passages from the cited paper, fused with BM25 and calibrated with isotonic regression on a held-out split. It is fast and cheap, and it is the direct analogue of the strongest existing citation checker, with 0.89 citation-level F1 on REALCITATIONERRORS-500 for bibliographic errors. Beating it means higher Macro-F1 and, above all, higher evidence sensitivity. We expect similarity scores to barely move between a factual rewrite and its original.

The fine-tuned competitor is a SciBERT cross-encoder that reads (assertion or context, retrieved passage) pairs and is trained on the benchmark's training split, with the rewrites used as extra training pairs. This is the reviewer's "small supervised model" answer. Its known weaknesses on subtle scientific text are 63.8 Macro-F1 on Athar polarity per TDCSA and 0.66 strict accuracy on MultiCite. It gets the same hyperparameter search budget as our trained span finder.

The strongest competitor is a vanilla frontier judge. Claude-3-7-sonnet and DeepSeek-R1 each receive the original citation context and the full text of the cited paper, with a chain-of-thought rubric. We also run an InnoEval-style variant with several reviewer personas that vote. All LLM systems, ours included, use the same underlying models, the same evidence access, and a matched token budget per item, with prompts tuned on the same development split. Beating this baseline means matching or exceeding its Macro-F1 while cutting the rhetorical flip rate and the popularity gap by at least half.

The ablations isolate the key ingredient. We run the full pipeline with the original rhetorical wording passed to the verifier instead of the neutral assertion. We run it with the cited paper's identity revealed. And we give the verifier the full paper instead of retrieved passages, to test whether narrow evidence helps or merely truncates.

Why not just…?

Why not just give a frontier LLM the citing context and the full cited paper? This is the real threat, and the experiment is designed to settle it. Long-context frontier models may already read papers well enough. What the literature questions is not their reading but their judgment: Articulate Intuition or Genuine Analysis? shows judges scoring style, and Curse of Knowledge shows reasoning models are the most easily biased. If the vanilla judge's rhetorical flip rate and popularity gap are already close to zero and its accuracy matches ours, we will report that the decomposition adds nothing and present the benchmark as a stress test. The ablation that passes original wording through our pipeline shows whether any gain comes from removing rhetoric rather than from retrieval.

Why not just use embedding similarity with calibration, as CiteGuard does? Similarity measures topic, and a false claim about a paper is usually as on-topic as a true one. The factual-rewrite pairs test this directly: if the similarity score does not separate an assertion from its minimally edited contradicted version, this approach cannot do the task. We expect it can still sort out the "not addressed" category and may be a useful cheap first filter, as it is in CiteGuard.

Why not just fine-tune SciBERT on the new labels? With about 1,500 human-labeled assertions, contradicted cases will be scarce. Prior results show small encoders struggle with implicit scientific evaluation even with more data. We run it anyway. If it matches the LLM pipeline on accuracy and invariance, it is the better deployable answer, and we would recommend it with LLM-generated rewrites as its training data.

Why not just classify sentiment and treat negative citations as suspect? Polarity is not fidelity. A correct, well-supported criticism and a misrepresentation are both negative. The polarity-only heuristic baseline measures exactly how far this proxy goes.

Proposed Experiments

The main table reports four-way Macro-F1, recall on the contradicted class, rhetorical flip rate, evidence sensitivity and the popularity gap for every system on CiteFidelity. Agreement with humans is reported as kappa and compared with inter-annotator kappa. The decisive check is paired. Our claim holds only if the vanilla frontier judge shows a clearly non-zero rhetorical flip rate and the decomposed pipeline reduces it without a drop in Macro-F1, with confidence intervals from bootstrap resampling over assertions. A second check examines the "not checkable" category: we expect vanilla judges to confuse it with "contradicted," because a confident outside critique sounds like a factual refutation. We will report that confusion cell directly. A third check tests transfer. We apply the pipeline, trained and tuned only on computational linguistics, to a small annotated sample of machine learning citations from the cs.LG papers used by Algaba et al., and to biomedical citations from PubMed Central papers in REALCITATIONERRORS-500. This addresses the single-discipline limitation noted for MultiCite and the Athar corpus. Finally, we run the full pipeline over the evaluative citations in a few hundred Anthology papers and hand-audit its highest-confidence "contradicted" verdicts. This yields a first estimate of how often published praise and criticism misstate the work they describe, and whether that rate differs between famous and obscure cited papers.