Statistical modelling of citation networks, research influence and journal prestige

David SelbyWarwick 2020

Abstract Standard approaches to measuring the ‘impact’ of academic journals or individual researchers are often not based on principled statistical methods. This research demonstrates the value of statistical modelling for citation data, reproducing quality judgements of human assessors more faithfully than traditional metrics. The study uncovers a deep theoretical connection between PageRank and the Bradley–Terry model, extends metrics to model interdisciplinary influence through ‘super-journals,’ and addresses the ‘ecological inference’ problem of estimating individual paper quality from aggregate national assessment data (REF 2014).

Introduction: Beyond the Journal Impact Factor

In the academic ecosystem, the "publish or perish" culture relies heavily on metrics to determine career progression and funding. However, the dominant tool—the Journal Impact Factor (JIF)—is a blunt instrument. As a simple arithmetic mean, it fails to account for skewed citation distributions, ignores the prestige of the citing source, and lacks a framework for uncertainty.

David Antony Selby’s research argues for a transition toward rigorous statistical frameworks. By treating citation networks as structured data rather than simple counts, we can bridge the gap between popular ranking algorithms and formal probability models, providing a clearer view of how scientific influence actually flows across disciplines.

Methodology: Bridging Algorithms and Statistics

The research employs several sophisticated techniques to move beyond heuristic rankings:

  • The Bradley–Terry (Stigler) Model: Traditionally used to rank sports teams, this model treats a citation from Journal A to Journal B as a "win" for B’s influence. This allows for the calculation of relative prestige with associated "error bars."
  • The Unified Theory of Ranking: A core contribution is the mathematical proof that Google’s PageRank and the Bradley–Terry model are linked. Under the condition of "quasi-symmetry" in a citation network, both methods yield identical rankings, effectively providing a statistical foundation for algorithmic scores.
  • The Scroogefactor: This metric identifies the most concentrated sources of prestige. It penalizes "generous" journals (those with long bibliographies, like review journals) and rewards "miserly" ones that cite sparingly, highlighting journals that export more influence than they consume.
  • Ecological Inference for REF 2014: To analyze the UK’s Research Excellence Framework (REF), where only aggregate departmental scores are public, Selby used Poisson binomial models to infer the underlying quality associated with specific journals and institutions.

Mapping Intellectual Influence

By aggregating publications into "super-journals" representing entire fields, the study maps the global flow of intellectual influence. The analysis reveals consistent "trade deficits" in citations: pure theoretical fields, such as Mathematics and Statistics, act as major exporters of influence toward applied fields like Medicine and Engineering.

Furthermore, the research introduces a methodology of residual diagnosis for network community structures. This allows researchers to evaluate how journals are grouped into fields by identifying "excess self-citation" that might otherwise distort influence models.

Key Findings from National Assessments

The application of these models to the REF 2014 dataset yielded significant insights into the relationship between journal prestige and human expert peer review:

  1. Predictive Accuracy: The ecological inference model predicted departmental REF scores with approximately 82% accuracy based solely on publication venues.
  2. The Human Signal: Despite the high correlation with journal prestige, the model identified a "shrinkage effect." High-prestige institutions often scored higher than their journal choices alone would predict, while lower-tier institutions scored lower. This suggests that human panels successfully identify the "best of the best" within a journal, proving that expert assessment captures a quality signal that simple journal-counting misses.
  3. Disciplinary Exporters: Statistics and Psychometrics were confirmed as vital "service" disciplines, providing the methodological backbone for a wide array of external research fields.

Conclusion and Future Directions

This work demonstrates that while journal prestige is a powerful proxy for quality, it is not a replacement for expert review. The statistical tools developed—including the scrooge and ref2014 R packages—provide a reproducible framework for future bibliometric analysis.

Future research may apply these diagnostic tools to detect "citation cartels"—groups of journals that artificially inflate metrics through mutual citation—and extend the models to track the dynamic evolution of field prestige over time. By moving toward principled statistical modelling, the academic community can ensure that impact metrics are as rigorous as the science they seek to measure.

Summary updated

Abstract

Standard approaches to measurement of the ‘impact’ of academic journals, or even sometimes of individual researchers or single research outputs, are typically not based on principled statistical methods for the analysis of citation data, through appropriate statistical models. Recent research has shown the value of such statistical modelling, for citations within a research discipline, for example in reproducing more faithfully the quality judgements of human assessors. In this project we study the strengths and weaknesses of statistical modelling approaches to citation-network data, and in so doing, uncover a deep theoretical connection between two otherwise unrelated journal ranking methods: PageRank and the Bradley–Terry model. 
\n
\nWe extend the usual journal- or author-based metrics, by aggregating all publications in a given field into ‘super-journals’. This permits modelling the exchange of citations between disciplines, raising the question: which scientific fields export the most intellectual influence, through recent research, to other fields? The relative merits of human and algorithmic field classifications are discussed. For this task, we propose a methodology of residual diagnosis for network community structures. 
\n
\nFinally, we investigate the extent to which the 2014 Research Excellence Framework’s assessment of ‘quality’ of research outputs (rated 4*, 3*, 2* or 1*) was associated with the reputation of the journals in which those outputs were published. Submissions data are available, as are the aggregate scores for each university department, but the individual ratings for each paper are not. The research question is thus an ‘ecological inference’ problem attempting to estimate individual-level characteristics from aggregate data. Results are presented for several research fields. 
\n
\nTo promote reproducibility and enable future research, the thesis includes a vignette on how to obtain citation network data from various databases, and is accompanied by R packages scrooge and ref2014 to facilitate analysis. 
\n
BibTeX
@inproceedings{selby2020statistical,
  author = {David Selby},
  title = {{Statistical modelling of citation networks, research influence and journal prestige}},
  booktitle = {Warwick},
  year = {2020},
}
Figure