CausalCite: A Causal Formulation of Paper Citations

Ishan Agrawal, Zhijing Jin, Ehsan Mokhtarian, Siyuan Guo, Yuen Chen, Mrinmaya Sachan, Bernhard Schölkopf

Introduction

Citation counts have long served as the primary metric for evaluating scientific impact. However, this "gold standard" is inherently flawed, often reflecting popularity, sub-field inflation, or institutional prestige rather than pure intellectual contribution. CausalCite introduces a paradigm shift by moving from simple counting to causal attribution.

The fundamental question CausalCite asks is counterfactual: "What would have happened to the success of a follow-up study if the original paper had never existed?" By isolating the specific impact a paper has on its successors, this framework provides a more equitable and accurate assessment of research quality.

The counterfactual approach to measuring paper impact: comparing the actual success of a follow-up study to a hypothetical scenario where the original paper was never published.

Methodology: The TEXTMATCH Framework

To operationalize this counterfactual logic, the authors developed TEXTMATCH, a method that adapts causal matching to high-dimensional text data.

The Causal Graph

The framework identifies the "treatment" ($T$) as the act of building a follow-up paper $b$ on paper $a$. The "effect" ($Y$) is the subsequent success of paper $b$. To accurately measure this effect, the model must control for "confounders" ($X$)—variables that influence both the likelihood of a citation and the success of the follow-up paper, such as the research topic or publication year.

The causal graph identifies confounders like topic and year that must be controlled for, while distinguishing them from mediators and colliders that should not be controlled.

Implementation

  1. Representation: Titles and abstracts are encoded into high-dimensional vectors using MPNet LLM embeddings to capture semantic content.
  2. Synthetic Counterfactuals: For every citing paper, TEXTMATCH identifies the ten most similar papers that did not cite the original. It then calculates a weighted average of their success metrics to create a robust synthetic counterfactual.
  3. Impact Indices:
    • PCI (Pairwise Causal Impact): The individual impact on one follow-up study.
  4. TCI (Total Causal Impact): The aggregate impact across all successors.
  5. ACI (Average Causal Impact): The normalized impact per follow-up paper.

Evaluation and Key Findings

The researchers validated CausalCite using the Semantic Scholar Academic Graph (206M papers) and specialized benchmarks, including author-identified influential citations and Test-of-Time (ToT) awards.

Superior Accuracy and Correlation

CausalCite achieved an 80.29% accuracy in identifying truly influential references, significantly outperforming raw citation counts (71.33%). More importantly, it showed a +30.14% improvement in correlating with prestigious Test-of-Time awards compared to raw citations. While "Best Paper" awards are often difficult to predict at the time of publication, CausalCite excels at retrospective evaluation, showing a much stronger correlation with long-term impact (0.64 vs 0.34).

Topic Invariance and Fairness

A critical advantage of CausalCite is its stability across disciplines. While raw citation counts in Computer Vision are often four times higher than in NLP, the Average Causal Impact (ACI) remains consistent. This suggests that CausalCite effectively mitigates the "popularity bias" and "citation inflation" that plague cross-domain comparisons.

Identifying Hidden Gems

The metric also helps identify "undercited" papers—those that serve as crucial technical baselines but lack the high raw citation numbers of more "trendy" publications. By analyzing the residual between citations and CausalCite, researchers can uncover works with high intellectual weight that might otherwise be overlooked.

Limitations and Future Directions

Despite its advantages, CausalCite faces several challenges:

  • Computational Intensity: The complex matching process requires significant resources, taking roughly 25 minutes per paper.
  • Cold Start: The metric requires existing citations to function, making it unsuitable for brand-new publications.
  • Scope of Content: Current models rely on abstracts and titles; incorporating full-text data could further refine the causal matching.
  • Credit Assignment: Future work is needed to determine how to distribute a paper's causal impact among its co-authors fairly.

Conclusion

CausalCite represents a significant step toward a more rigorous and fair evaluation of scientific progress. By grounding impact in the principles of causal inference, it offers a tool that values intellectual contribution over mere popularity.

Summary updated

Abstract

Citation count of a paper is a commonly used proxy for evaluating the significance of a paper in the scientific community. Yet citation measures are widely criticized for failing to accurately reflect the true impact of a paper. Thus, we propose CausalCite, a new way to measure the significance of a paper by assessing the causal impact of the paper on its follow-up papers. CausalCite is based on a novel causal inference method, TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings. TextMatch encodes each paper using text embeddings from large language models (LLMs), extracts similar samples by cosine similarity, and synthesizes a counterfactual sample as the weighted average of similar papers according to their similarity values. We demonstrate the effectiveness of CausalCite on various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, (test-of-time) awards for past papers, and its stability across various subfields of AI. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of the quality of a paper. Our code is available at https://github.com/causalNLP/causal-cite.
BibTeX
@inproceedings{agrawal2024causalcite,
  author = {Ishan Agrawal and Zhijing Jin and Ehsan Mokhtarian and Siyuan Guo and Yuen Chen and Mrinmaya Sachan and Bernhard Schölkopf},
  title = {{CausalCite: A Causal Formulation of Paper Citations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.497},
}
Figure