TL;DR
CausalCite is a new metric that moves beyond raw citation counts by using causal inference to measure a paper's true impact. By leveraging Large Language Models (LLMs) to synthesize "counterfactual" scenarios—asking what would have happened to a follow-up study if it hadn't cited the original work—it provides a more equitable assessment that is stable across different research fields and aligns better with expert judgment.
What Problem Does This Paper Address?
In academia, citation counts are the "gold standard" for impact, yet everyone knows they are deeply flawed. They are easily gamed, biased toward established researchers or specific countries, and vary wildly between fields (a top-tier Computer Vision paper might naturally get 4x the citations of an equally influential NLP paper).
The core issue is that a citation doesn't always imply causal impact. A paper might be cited simply because it's in a "hot" topic, or as a generic mention in a related work section. The authors of CausalCite want to isolate the actual contribution: How much of a follow-up paper's success is truly attributable to the paper it cited?
Methods
The authors frame paper impact as a causal inference problem. They define the Pairwise Causal Impact (PCI) as the difference between the success of a follow-up paper $b$ (measured by $\log_{10}$ citations) and its "counterfactual" success if it hadn't built upon paper $a$.
To estimate this unobservable counterfactual, they introduce TEXTMATCH:
- Encoding: They use an LLM (specifically MPNet) to turn paper titles and abstracts into high-dimensional embeddings.
- Matching: For a follow-up paper $b$, they find a "control group" of papers published in the same year that address similar topics but do not cite paper $a$.
- Synthesis: Instead of a simple 1-to-1 match, they create a synthetic counterfactual by taking a weighted average of the top 10 most similar control papers. This stabilizes the estimate against the high variance of citation counts.
- Scaling: To handle the massive scale of the Semantic Scholar graph, they use a two-stage retrieval process: fast BM25 keyword matching followed by fine-grained neural reranking.
Datasets
- Semantic Scholar Academic Graph: The primary source, containing 206M papers and 2.4B citation edges.
- Expert-Annotated Impact Dataset: 1,037 papers where authors manually identified which of their references actually had a significant impact on their work.
- Test-of-Time (ToT) Awards: A curated set of 792 papers (72 ToT winners and 720 controls) from top AI, NLP, and CV conferences.
- Best Paper Awards: 444 papers used to compare "forecasting" impact vs. retrospective causal impact.
Key Insights and Ideas
- Topic Invariance: This is perhaps the most significant finding. While raw citation counts for Test-of-Time papers in Computer Vision are much higher than in NLP, the Average Causal Impact (ACI) remains consistent across fields. This suggests CausalCite is a much fairer "cross-field" metric.
- The "Best Paper" Paradox: CausalCite correlates strongly with Test-of-Time awards (0.639) but much more weakly with "Best Paper" awards (0.348). This highlights that predicting future impact at the time of publication is incredibly difficult, even for expert committees.
- Identifying Undercited Gems: By looking at the residuals between citations and CausalCite, the authors can find "undercited" papers—foundational works that act as critical baselines but haven't racked up massive raw numbers. These papers are often cited heavily in the results sections of follow-up work, indicating their technical importance.
Key Results
- Expert Alignment: CausalCite achieved 80.29% accuracy in identifying "significant" citations as judged by experts, outperforming raw citations (71.33%) and Semantic Scholar’s "Highly Influential Citations" (75.25%).
- ToT Correlation: The metric showed a 30% higher correlation with prestigious Test-of-Time awards compared to traditional citation counts.
- Distribution: As shown in the violin plots, CausalCite values for Test-of-Time papers are significantly higher and more distinct from non-winners than raw citation counts would suggest.
Limitations and Future Work
- Computational Cost: Calculating the metric is expensive, taking roughly 25 minutes per paper-pair evaluation due to the complex matching and reranking required.
- Data Lag: The system relies on Semantic Scholar data, which can lag behind Google Scholar for the most recent publications.
- Textual Scope: Currently, the model only "sees" titles and abstracts. Future versions could benefit from analyzing the full text of papers to better understand the context of a citation.
- Zero-Citation Cold Start: Like any citation-based metric, it cannot evaluate a brand-new paper with zero citations.
Summary updated