The Role of Positive and Negative Citations in Scientific Evaluation

Xiaomei Bai, Ivan Lee, Zhaolong Ning, Amr Tolba, Feng Xia

TL;DR

Not all citations are created equal, yet most academic ranking systems treat them as identical. This paper introduces PANDORA, an algorithm that identifies "Conflict of Interest" (COI) citations—such as those between frequent collaborators or colleagues—and weights them based on their objective relevance to ensure a fairer, more accurate evaluation of scientific impact.

What Problem Does This Paper Address?

In the "publish or perish" world of academia, citation counts are the primary currency for funding, hiring, and rankings. However, this system is prone to "gaming." Researchers often engage in excessive self-citation or form "citation circles" with collaborators to artificially inflate their impact.

While some self-citations are a legitimate part of the research narrative, others are purely promotional. Current metrics like the h-index or standard PageRank-based rankings fail to distinguish between a citation from an independent peer and one from a close office mate. The challenge is identifying which COI citations are "positive" (scientifically necessary) and which are "negative" (potentially biased) without a universal "ground truth" to guide the way.

Methods

The authors propose the PANDORA (Positive And Negative COI-Distinguished Objective Rank Algorithm). Its workflow consists of three main pillars:

  1. COI Identification & Classification: The algorithm looks at the relationship between the citing and cited authors. It defines four categories:
  2. Positive COI: Authors have collaborated before, and their papers are frequently co-cited by independent third parties (indicating legitimate shared relevance).
  3. Negative COI: Authors have collaborated, but no independent papers cite them together (suggesting an insular citation).
  4. Suspected COI (Positive/Negative): Authors belong to the same institution but haven't collaborated. The "Positive/Negative" distinction again relies on whether independent third parties co-cite their work.
  5. Dynamic Weighting: Instead of giving every citation a weight of 1, PANDORA applies an exponential decay formula to "Negative" COI citations. The weight is reduced based on collaboration frequency, time spans, and citation history. "Positive" COIs are treated as standard citations.
  6. Integrated Ranking: PANDORA combines a weighted PageRank (for paper impact) with the HITS algorithm (to evaluate the "authority" of authors, journals, and references). Crucially, it uses a credit allocation algorithm to fairly distribute a paper's impact among its co-authors rather than giving full credit to everyone.

Datasets

The study utilized a massive dataset from the American Physical Society (APS), specifically focusing on the Physical Review C and Physical Review E subsets.

  • Scope: ~71,287 publications spanning 1970 to 2013 (43 years).
  • Data Points: Titles, DOIs, author lists, publication dates, affiliations, and a comprehensive citation graph.

Key Insights and Ideas

  • The Co-Citation Litmus Test: The most clever part of PANDORA is using "co-citation" as a proxy for legitimacy. If the rest of the scientific community thinks two papers belong together (by citing them in the same breath), a citation between those authors is likely valid. If only the authors themselves link their papers, it’s flagged as a negative COI.
  • The "Big Player" Bias: The researchers found that larger institutions and countries naturally have higher COI citation counts. This isn't necessarily "cheating"—large organizations simply have more internal collaboration. However, without PANDORA, this volume can drown out high-quality work from smaller, less-connected labs.
  • The 1993 Surge: The data showed a sudden spike in citation impact and COI citations starting around 1993. The authors attribute this to the "Information Superhighway" policy in the US and the rise of the internet, which fundamentally changed how researchers collaborate and cite.

Key Results

  • Superior Accuracy: PANDORA consistently outperformed existing state-of-the-art methods like FutureRank and CAJTRank.
  • Recommendation Intensity (RI): In "Top-K" paper rankings, PANDORA achieved accuracy rates between 0.656 and 0.8, significantly higher than its competitors (which hovered between 0.5 and 0.67).
  • Correlation: It showed a much stronger Spearman’s rank correlation with the ground truth (citations filtered for COI), proving it better reflects "true" scientific influence.
  • Geographic Leaders: The analysis identified the US, Germany, and Israel as leaders in research impact, while also noting they have the highest volumes of COI citations due to their massive internal research infrastructures.

Limitations and Future Work

  • Disciplinary Silo: The study was limited to the physics community (APS dataset). Citation behaviors vary wildly between fields (e.g., social sciences vs. high-energy physics), so the model may need recalibration for other disciplines.
  • Hidden Ties: PANDORA identifies COIs through co-authorship and shared affiliations. It cannot currently detect "hidden" conflicts of interest, such as personal friendships, mentor-student relationships where the mentor isn't a co-author, or anonymous peer-review biases.
  • Data Quality: The algorithm relies on how publishers (like APS) record affiliations. If a university uses multiple names or variants, the "same institution" detection might fail. Future work could involve more robust entity resolution for authors and institutions.

Summary updated

Abstract

Quantifying the impact of scientific papers objectively is crucial for research output assessment, which subsequently affects institution and country rankings, research funding allocations, academic recruitment and national/international scientific priorities. While most of the assessment schemes based on publication citations may potentially be manipulated through negative citations, in this study, we explore Conflict of Interest (COI) relationships and discover negative citations and subsequently weaken the associated citation strength. PANDORA (Positive And Negative COI- Distinguished Objective Rank Algorithm) has been developed, which captures the positive and negative COI, together with the positive and negative suspected COI relationships. In order to alleviate the influence caused by negative COI relationship, collaboration times, collaboration time span, citation times and citation time span are employed to determine the citing strength; while for positive COI relationship, we regard it as normal citation relationship. Furthermore, we calculate the impact of scholarly papers by PageRank and HITS algorithms, based on a credit allocation algorithm which is utilized to assess the impact of institutions fairly and objectively. Experiments are conducted on the publication dataset from American Physical Society (APS) dataset, and the results demonstrate that our method significantly outperforms the current solutions in Recommendation Intensity of list R at top-K and Spearman's rank correlation coefficient at top-K.
BibTeX
@article{bai2017role,
  author = {Xiaomei Bai and Ivan Lee and Zhaolong Ning and Amr Tolba and Feng Xia},
  title = {{The Role of Positive and Negative Citations in Scientific Evaluation}},
  journal = {IEEE Access},
  year = {2017},
  doi = {10.1109/access.2017.2740226},
}
Figure