Judging a book by its cover: how much of REF `research quality' is really `journal prestige'?

David Selby, David FirtharXiv 2023 · DOI · arXiv

Abstract

The Research Excellence Framework (REF) is the UK’s primary system for assessing research quality in higher education, directly influencing the distribution of billions in funding. While the list of submitted outputs is public, the individual scores assigned by expert panels remain confidential, creating a "black box" that fuels suspicion: do panels judge research based on its intrinsic quality or the prestige of the journal in which it appears? This study addresses this question by framing the REF assessment as an "ecological inference" problem. Using logit models and regularization techniques on REF 2014 data, the researchers quantify the extent to which journal identity predicts quality profiles across fields like Economics, Physics, and Chemistry.

Introduction: The "Black Box" of Research Assessment

In the UK academic landscape, the REF is a high-stakes exercise. Expert panels rate research outputs on a scale from 1 (nationally recognized) to 4 (world-leading). However, the transparency of this process is limited. While the public can see which papers a department submitted, they only see the aggregate "quality profile"—the percentage of papers at each star level.

This lack of granular data has led to a persistent debate: are panels truly reading the papers, or are they using journal titles as a proxy for quality? If a paper appears in Nature, is it an automatic 4*? This research aims to crack open this box by statistically inferring individual paper ratings from group-level results.

The Ecological Inference Challenge

The primary statistical hurdle in this analysis is the "ecological inference" problem. This occurs when one tries to draw conclusions about individual-level behavior based on aggregate data. In the context of the REF, we know the "marginal" totals (how many papers were in which journal, and how many total 4 ratings a department received), but we do not know the internal cell counts (how many 4 ratings went to papers in specific journals).

The classic ecological inference problem: inferring individual-level behavior (e.g., voting or paper ratings) from aggregate marginal totals.

As illustrated in the figure above, this is analogous to trying to determine how different demographic groups (e.g., Black vs. White voters) voted in a secret ballot based only on precinct-level totals ($T_i$). In this study, the "voters" are the research outputs, and the "candidates" are the quality ratings.

Methodology

To solve this, the authors modeled the probability of a paper receiving a 4* rating as a function of its publication venue. They employed several advanced techniques:

  • Poisson-Binomial Model: This treats each submitted paper as an independent trial where the "success" probability is determined by the journal's inferred prestige.
  • Estimation Paths: The team used Hamiltonian Monte Carlo (HMC) via the Stan programming language for a Bayesian perspective, providing uncertainty intervals, and an Expectation-Maximization (EM) algorithm for frequentist maximum-likelihood estimation.
  • Regularization: To handle journals with very few submissions, they used "pseudo-data" (priors) to pull estimates toward a global mean, ensuring that small sample sizes didn't produce extreme, unrealistic quality scores.

The study focused on four "Units of Assessment" from REF 2014: Economics & Econometrics, Mathematical Sciences, Physics, and Chemistry, involving over 20,000 total outputs.

Key Insights: Prestige vs. Performance

The model's findings offer a nuanced view of the REF process.

The "Top Five" and Journal Hierarchy

In Economics, the inferred rankings perfectly mirrored the discipline’s legendary "Top Five" journals. If a paper was published in one of these venues, its probability of receiving a 4* rating was near 100%. This confirms that in certain fields, journal prestige is an incredibly strong predictor of REF success.

The "Shrinkage Effect": Panels Do Read

Perhaps the most significant finding is what the authors call the "shrinkage effect." The model revealed that top-tier institutions (like Oxford or Cambridge) consistently received higher scores than the model predicted based on their journal profiles alone. Conversely, lower-ranked institutions often performed worse than their journal lists suggested.

This is actually a positive sign for the REF's integrity. It suggests that panels are not just "judging a book by its cover." Instead, they appear to be distinguishing the truly exceptional papers within high-tier journals from the more standard ones, and identifying high-quality work even when it appears in less prestigious venues.

Impact Factor is a Blunt Instrument

The study also highlighted the limitations of generic metrics. In Mathematical Sciences, there was almost zero correlation between a journal’s Impact Factor and its inferred REF quality. This underscores the danger of using one-size-fits-all metrics to judge specialized research fields.

Results and Impact

The model demonstrated high predictive accuracy. The "index of dissimilarity" ($\Delta$)—the proportion of papers that would need to be reclassified to match actual REF results—ranged from a low of 8.2% in Chemistry to 17.9% in Economics.

Furthermore, the authors calculated the "redistribution of monetary reward" ($\Delta_£$). They found that if funding were allocated based solely on journal identity rather than the actual, panel-assessed REF scores, approximately 5.6% to 8.9% of the total research budget would shift between institutions. This suggests that while journal prestige is a major factor, the expert panels' "value-add" significantly alters the final financial outcomes for universities.

Conclusion

While journal prestige is undeniably a massive predictor of a department's success in the REF, it is not the only factor. The "shrinkage effect" provides evidence that expert panels are looking past the journal title to assess the intrinsic quality of the work. However, the high correlation in fields like Economics suggests that the "prestige" of a venue remains a powerful, if not dominant, signal in research assessment.

Summary updated

Abstract

The Research Excellence Framework (REF) is a periodic UK-wide assessment of the quality of published research in universities. The most recent REF was in 2014, and the next will be in 2021. The published results of REF2014 include a categorical `quality profile' for each unit of assessment (typically a university department), reporting what percentage of the unit's REF-submitted research outputs were assessed as being at each of four quality levels (labelled 4*, 3*, 2* and 1*). Also in the public domain are the original submissions made to REF2014, which include -- for each unit of assessment -- publication details of the REF-submitted research outputs. In this work, we address the question: to what extent can a REF quality profile for research outputs be attributed to the journals in which (most of) those outputs were published? The data are the published submissions and results from REF2014. The main statistical challenge comes from the fact that REF quality profiles are available only at the aggregated level of whole units of assessment: the REF panel's assessment of each individual research output is not made public. Our research question is thus an `ecological inference' problem, which demands special care in model formulation and methodology. The analysis is based on logit models in which journal-specific parameters are regularized via prior `pseudo-data'. We develop a lack-of-fit measure for the extent to which REF scores appear to depend on publication venues rather than research quality or institution-level differences. Results are presented for several research fields.
BibTeX
@misc{selby2023judging,
  author = {David Selby and David Firth},
  title = {{Judging a book by its cover: how much of REF `research quality' is really `journal prestige'?}},
  year = {2023},
  eprint = {2306.13367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.13367},
}
Figure