Empirical Bayes approaches to PageRank type algorithms for rating scientific journals

Jean‐Louis Foulley, Gilles Celeux, Julie JossearXiv 2017 · DOI · arXiv

Introduction

The evaluation of scientific journals has long been dominated by the Journal Impact Factor (IF). However, the IF has faced significant criticism for being a simplistic average that fails to account for the "prestige" of the citing source and is susceptible to manipulation through self-citations. In response, modern influence scores such as the Eigenfactor (EIFA) and the Prestige Scimago Journal Rank (PSJR) have emerged. These metrics utilize PageRank-style algorithms on citation networks, weighing citations based on the influence of the journal providing them.

While these network-based metrics are an improvement, they rely on a "teleportation" mechanism—a damping factor (typically fixed at $\alpha = 0.85$)—to ensure the algorithm converges. This paper provides a rigorous statistical foundation for these scores by reinterpreting the PageRank smoothing matrix as the posterior mean of a Dirichlet-multinomial model within an Empirical Bayes framework.

The Problem: Arbitrariness and Zero-Handling

Traditional PageRank-based metrics suffer from two primary theoretical gaps:

  1. Arbitrary Damping: The choice of $\alpha = 0.85$ is a heuristic carried over from web search, applied uniformly to every journal regardless of its citation volume or behavior.
  2. Zero-Handling: Citation matrices are sparse. Standard algorithms often fail to distinguish between a "sampling zero" (a journal that could have been cited but wasn't in this specific window) and a "structural zero" (a journal that cannot be cited, such as when self-citations are intentionally excluded to prevent bias).

The Empirical Bayes Framework

The authors propose a probabilistic model that replaces the global damping factor with a data-driven, journal-specific parameter. By modeling the citation process through a Dirichlet-multinomial framework, the "random jump" or teleportation in PageRank is revealed to be mathematically equivalent to a Bayesian prior.

Journal-Specific Damping

Instead of a fixed $\alpha$, the model derives a shrinkage coefficient $\alpha_i$ for each journal $i$. This factor is determined by the number of references a journal produces ($n_i$) and a concentration parameter ($K$). For journals with sparse data (dangling nodes), the model automatically relies more heavily on the prior (teleportation). For journals with high citation volumes, the model allows the data to dominate the score.

Distinguishing Zeros

By explicitly modifying the Dirichlet prior, the authors can handle structural zeros rigorously. This is particularly important for managing self-citations. In many metrics, self-citations are simply removed, but this can bias the remaining probability distribution. The Empirical Bayes approach allows for the exclusion of self-citations without distorting the underlying statistical model.

The EBEF Algorithm

The authors introduce the Empirical Bayes Eigen Factor (EBEF). To estimate the model's hyperparameters, they employ a Majorize-Minimize (MM) algorithm. Specifically, a fixed-point iteration is used to maximize the marginal likelihood of the citation data. This process yields a concentration parameter ($K$) that reflects the actual density and behavior of the specific citation network being analyzed, rather than relying on a pre-set constant.

Case Study: 47 Statistical Journals

The methodology was tested using a cross-citation matrix of 47 statistical journals (data from Varin et al., 2016), covering citations from 2010 for articles published between 2001 and 2010. The dataset included prestigious titles like JASA and Annals of Statistics, alongside software-centric journals like the Journal of Statistical Software (JSS).

Key Findings

  • High Correlation: The EBEF scores showed a Spearman rank correlation of approximately 0.99 with the standard Eigenfactor, validating the Bayesian approach as a theoretically sound alternative that reaches similar conclusions to established empirical methods.
  • Stability of the Elite: The "top quintet" of journals—JRSS-B, STSCI, AOS, JASA, and BKA—remained stable across different scoring methods, indicating that the highest-tier journals are robust to algorithmic variations.
  • Self-Citation Nuance: The model highlighted the impact of self-citations in software journals. For instance, the Stata Journal (STATAJ) has a self-citation rate of 67%. The EBEF framework allows for a more nuanced adjustment of these outliers than traditional methods.
  • Algorithm Efficiency: The fixed-point iteration proved highly efficient, reaching parameter estimation in 5–7 seconds, while the Levenberg-Marquardt algorithm provided useful asymptotic variance estimates.

Conclusion and Future Directions

This research demonstrates that the "smoothing" required for journal rankings is not just a computational necessity but a statistical one that can be optimized using Empirical Bayes. By moving from a fixed damping factor to a journal-specific shrinkage coefficient, the EBEF provides a more tailored and theoretically grounded ranking system.

Future work may move away from the radical exclusion of self-citations toward a "data-driven underweighting" approach. This would involve a tuning parameter to attenuate self-citations based on field-specific norms, ensuring that journals are neither unfairly penalized nor artificially inflated by their own citation practices. Additionally, expanding the model to handle cross-disciplinary networks remains a key area for further development.

Summary updated

Abstract

Following criticisms against the journal Impact Factor, new journal influence scores have been developed such as the Eigenfactor or the Prestige Scimago Journal Rank. They are based on PageRank type algorithms on the cross-citations transition matrix of the citing-cited network. The PageRank algorithm performs a smoothing of the transition matrix combining a random walk on the data network and a teleportation to all possible nodes with fixed probabilities (the damping factor being $α= 0.85$). We reinterpret this smoothing matrix as the mean of a posterior distribution of a Dirichlet-multinomial model in an empirical Bayes perspective. We suggest a simple yet efficient way to make a clear distinction between structural and sampling zeroes. This allows us to contrast cases with self-citations included or excluded to avoid overvalued journal bias. We estimate the model parameters by maximizing the marginal likelihood with a Majorize-Minimize algorithm. The procedure ends up with a score similar to the PageRank ones but with a damping factor depending on each journal. The procedures are illustrated with an example about cross-citations among 47 statistical journals studied by Varin et. al. (2016).
BibTeX
@misc{foulley2017empirical,
  author = {Jean‐Louis Foulley and Gilles Celeux and Julie Josse},
  title = {{Empirical Bayes approaches to PageRank type algorithms for rating scientific journals}},
  year = {2017},
  eprint = {1707.09508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.09508},
}
Figure