CausalCite: A Causal Formulation of Paper Citations

Ishan Kumar, Zhijing Jin, Ehsan Mokhtarian, Siyuan Guo, Yuen Chen, Mrinmaya Sachan, Bernhard Schölkopf

TL;DR

CausalCite is a new metric that moves beyond raw citation counts by using causal inference to measure a paper's true impact. By leveraging Large Language Models (LLMs) to synthesize "counterfactual" scenarios—asking what would have happened to a follow-up study if it hadn't cited the original work—it provides a more equitable assessment that is stable across different research fields and aligns better with expert judgment.

What Problem Does This Paper Address?

In academia, citation counts are the "gold standard" for impact, yet everyone knows they are deeply flawed. They are easily gamed, biased toward established researchers or specific countries, and vary wildly between fields (a top-tier Computer Vision paper might naturally get 4x the citations of an equally influential NLP paper).

The core issue is that a citation doesn't always imply causal impact. A paper might be cited simply because it's in a "hot" topic, or as a generic mention in a related work section. The authors of CausalCite want to isolate the actual contribution: How much of a follow-up paper's success is truly attributable to the paper it cited?

Figure 1: An overview of our research question.

Methods

The authors frame paper impact as a causal inference problem. They define the Pairwise Causal Impact (PCI) as the difference between the success of a follow-up paper $b$ (measured by $\log_{10}$ citations) and its "counterfactual" success if it hadn't built upon paper $a$.

To estimate this unobservable counterfactual, they introduce TEXTMATCH:

  1. Encoding: They use an LLM (specifically MPNet) to turn paper titles and abstracts into high-dimensional embeddings.
  2. Matching: For a follow-up paper $b$, they find a "control group" of papers published in the same year that address similar topics but do not cite paper $a$.
  3. Synthesis: Instead of a simple 1-to-1 match, they create a synthetic counterfactual by taking a weighted average of the top 10 most similar control papers. This stabilizes the estimate against the high variance of citation counts.
  4. Scaling: To handle the massive scale of the Semantic Scholar graph, they use a two-stage retrieval process: fast BM25 keyword matching followed by fine-grained neural reranking.
Figure 2: The causal graph used to identify confounders (like topic and year) that must be controlled for, while avoiding mediators (like performance) that would block the effect being measured.

Datasets

  • Semantic Scholar Academic Graph: The primary source, containing 206M papers and 2.4B citation edges.
  • Expert-Annotated Impact Dataset: 1,037 papers where authors manually identified which of their references actually had a significant impact on their work.
  • Test-of-Time (ToT) Awards: A curated set of 792 papers (72 ToT winners and 720 controls) from top AI, NLP, and CV conferences.
  • Best Paper Awards: 444 papers used to compare "forecasting" impact vs. retrospective causal impact.

Key Insights and Ideas

  • Topic Invariance: This is perhaps the most significant finding. While raw citation counts for Test-of-Time papers in Computer Vision are much higher than in NLP, the Average Causal Impact (ACI) remains consistent across fields. This suggests CausalCite is a much fairer "cross-field" metric.
  • The "Best Paper" Paradox: CausalCite correlates strongly with Test-of-Time awards (0.639) but much more weakly with "Best Paper" awards (0.348). This highlights that predicting future impact at the time of publication is incredibly difficult, even for expert committees.
  • Identifying Undercited Gems: By looking at the residuals between citations and CausalCite, the authors can find "undercited" papers—foundational works that act as critical baselines but haven't racked up massive raw numbers. These papers are often cited heavily in the results sections of follow-up work, indicating their technical importance.
Figure 5: The distribution of TCI values by percentile of 100 random papers, which shows a long tail indicating that high impact is concentrated in a relatively small portion of papers.

Key Results

  • Expert Alignment: CausalCite achieved 80.29% accuracy in identifying "significant" citations as judged by experts, outperforming raw citations (71.33%) and Semantic Scholar’s "Highly Influential Citations" (75.25%).
  • ToT Correlation: The metric showed a 30% higher correlation with prestigious Test-of-Time awards compared to traditional citation counts.
  • Distribution: As shown in the violin plots, CausalCite values for Test-of-Time papers are significantly higher and more distinct from non-winners than raw citation counts would suggest.
Figure 3: Distributions of ToT (mean: 142) and non-ToT pa- pers (mean: 1,623).

Limitations and Future Work

  • Computational Cost: Calculating the metric is expensive, taking roughly 25 minutes per paper-pair evaluation due to the complex matching and reranking required.
  • Data Lag: The system relies on Semantic Scholar data, which can lag behind Google Scholar for the most recent publications.
  • Textual Scope: Currently, the model only "sees" titles and abstracts. Future versions could benefit from analyzing the full text of papers to better understand the context of a citation.
  • Zero-Citation Cold Start: Like any citation-based metric, it cannot evaluate a brand-new paper with zero citations.

Summary updated

Abstract

Citation count of a paper is a commonly used proxy for evaluating the significance of a paper in the scientific community. Yet citation measures are widely criticized for failing to accurately reflect the true impact of a paper. Thus, we propose CausalCite, a new way to measure the significance of a paper by assessing the causal impact of the paper on its follow-up papers. CausalCite is based on a novel causal inference method, TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings. TextMatch encodes each paper using text embeddings from large language models (LLMs), extracts similar samples by cosine similarity, and synthesizes a counterfactual sample as the weighted average of similar papers according to their similarity values. We demonstrate the effectiveness of CausalCite on various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, (test-of-time) awards for past papers, and its stability across various subfields of AI. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of the quality of a paper. Our code is available at https://github.com/causalNLP/causal-cite.
BibTeX
@inproceedings{kumar2024causalcite,
  author = {Ishan Kumar and Zhijing Jin and Ehsan Mokhtarian and Siyuan Guo and Yuen Chen and Mrinmaya Sachan and Bernhard Schölkopf},
  title = {{CausalCite: A Causal Formulation of Paper Citations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.497},
}
Figure