Reply to Bakker et al.

Sean C. Rife, Joshua M. Nicholson, Ashish Uppala, Domenic RosatiHypothesis 2025 · DOI

Abstract

Bakker, Theis-Mahon, and Brown recently presented an analysis of citations in scite.ai, a citation analysis platform. They concluded that the Scite algorithm inaccurately classified citations, particularly those they regarded as supporting previous work. While we strongly believe that independent assessments of Scite’s classifications are valuable, we argue that Bakker et al.’s assessment is incorrect, and that their conclusions are due to their definition of what constitutes supporting or contrasting citations. Additionally, Bakker, et al. restricted their analyses to citations of retracted works in systematic literature reviews, which artificially limits the types of statements that could be considered supporting or contrasting a specific claim. In our reply, we document the rationale for Scite’s classification scheme. We also provide examples of how Scite classifies different types of citations, as well as how these classifications differ from those presented in Bakker et al.

Introduction

Scite.ai is a popular tool that uses deep learning to classify academic citations into three categories: supporting, contrasting, or mentioning. Recently, a paper by Bakker et al. evaluated Scite's performance on a dataset of systematic reviews citing retracted papers, concluding that Scite's accuracy was poor.

This reply addresses a classic machine learning evaluation problem: label mismatch. The Scite team argues that Bakker et al. used a completely different conceptual rubric to grade the model than the one used to train it. Bakker et al. defined a citation as "supporting" if it cited a retracted paper without mentioning the retraction, and "contrasting" if it explicitly noted the retraction. We argue this measures an author's awareness of a retraction, not whether the citing paper actually provides primary scientific evidence to support or refute the cited claims.

Defining Citation Classifications

Scite’s model is trained to look for scientific arguments or experimental facts. To be classified as "supporting" or "contrasting," the citing paper must present new evidence or reproducibility results that directly agree or disagree with the cited work. This is a fundamental distinction from sentiment analysis or administrative tracking (like retraction status).

Examples of Scite's citation classification scheme and the rationale for each category.

As shown in the examples above, Scite requires specific evidentiary links to move a classification beyond "mentioning." For instance, even a statement that sounds supportive but merely establishes connections between observations without providing primary evidence is classified as "mentioning."

The Label Mismatch Problem

The core of the disagreement lies in two key conceptual differences:

  • Retraction is not equivalent to Scientific Disagreement: Acknowledging that a cited paper has been retracted is an administrative or methodological note. It does not mean the citing authors are presenting primary experimental evidence that contrasts with the cited work. Scite correctly labels these as "mentioning," while the critics penalized Scite for not labeling them as "contrasting."
  • Ignorance is not equivalent to Scientific Support: If an author cites a retracted paper without realizing it is retracted, they aren't necessarily providing new evidence to support it. They are usually just referencing it for background context. Scite labels these as "mentioning," but the critics penalized Scite for not labeling them as "supporting."

You cannot accurately assess an ML model's performance if your ground-truth test set uses definitions that fundamentally contradict the model's training data. Sentiment analysis (positive/negative tone) is not the same as evidentiary classification (supporting/contrasting data).

Conclusion

While third-party evaluations of Scite.ai are highly valuable and necessary for continual improvement, they must use a classification scheme that resembles the definitions the model was trained on. Using proxy metrics like retraction status to evaluate evidentiary support creates false negatives and obscures the model's actual utility in identifying scientific reproducibility.

Summary updated

Abstract

Bakker, Theis-Mahon, and Brown1 recently presented an analysis of citations in scite.ai, a citation analysis platform. They concluded that the Scite algorithm inaccurately classified citations, particularly those they regarded as supporting previous work. While we strongly believe that independent assessments of Scite’s classifications are valuable, we argue that Bakker et al.’s assessment is incorrect, and that their conclusions are due to their definition of what constitutes supporting or contrasting citations. Additionally, Bakker, et al. restricted their analyses to citations of retracted works in systematic literature reviews, which artificially limits the types of statements that could be considered supporting or contrasting a specific claim. In our reply, we document the rationale for Scite’s classification scheme. We also provide examples of how Scite classifies different types of citations, as well as how these classifications differ from those presented in Bakker et al.
BibTeX
@inproceedings{rife2025reply,
  author = {Sean C. Rife and Joshua M. Nicholson and Ashish Uppala and Domenic Rosati},
  title = {{Reply to Bakker et al.}},
  booktitle = {Hypothesis},
  year = {2025},
  doi = {10.18060/28018},
}
Figure