Abstract
The Research Excellence Framework (REF) is the UK’s primary system for assessing research quality in higher education, directly influencing the distribution of billions in funding. While the list of submitted outputs is public, the individual scores assigned by expert panels remain confidential, creating a "black box" that fuels suspicion: do panels judge research based on its intrinsic quality or the prestige of the journal in which it appears? This study addresses this question by framing the REF assessment as an "ecological inference" problem. Using logit models and regularization techniques on REF 2014 data, the researchers quantify the extent to which journal identity predicts quality profiles across fields like Economics, Physics, and Chemistry.
Introduction: The "Black Box" of Research Assessment
In the UK academic landscape, the REF is a high-stakes exercise. Expert panels rate research outputs on a scale from 1 (nationally recognized) to 4 (world-leading). However, the transparency of this process is limited. While the public can see which papers a department submitted, they only see the aggregate "quality profile"—the percentage of papers at each star level.
This lack of granular data has led to a persistent debate: are panels truly reading the papers, or are they using journal titles as a proxy for quality? If a paper appears in Nature, is it an automatic 4*? This research aims to crack open this box by statistically inferring individual paper ratings from group-level results.
The Ecological Inference Challenge
The primary statistical hurdle in this analysis is the "ecological inference" problem. This occurs when one tries to draw conclusions about individual-level behavior based on aggregate data. In the context of the REF, we know the "marginal" totals (how many papers were in which journal, and how many total 4 ratings a department received), but we do not know the internal cell counts (how many 4 ratings went to papers in specific journals).
As illustrated in the figure above, this is analogous to trying to determine how different demographic groups (e.g., Black vs. White voters) voted in a secret ballot based only on precinct-level totals ($T_i$). In this study, the "voters" are the research outputs, and the "candidates" are the quality ratings.
Methodology
To solve this, the authors modeled the probability of a paper receiving a 4* rating as a function of its publication venue. They employed several advanced techniques:
- Poisson-Binomial Model: This treats each submitted paper as an independent trial where the "success" probability is determined by the journal's inferred prestige.
- Estimation Paths: The team used Hamiltonian Monte Carlo (HMC) via the Stan programming language for a Bayesian perspective, providing uncertainty intervals, and an Expectation-Maximization (EM) algorithm for frequentist maximum-likelihood estimation.
- Regularization: To handle journals with very few submissions, they used "pseudo-data" (priors) to pull estimates toward a global mean, ensuring that small sample sizes didn't produce extreme, unrealistic quality scores.
The study focused on four "Units of Assessment" from REF 2014: Economics & Econometrics, Mathematical Sciences, Physics, and Chemistry, involving over 20,000 total outputs.
Key Insights: Prestige vs. Performance
The model's findings offer a nuanced view of the REF process.
The "Top Five" and Journal Hierarchy
In Economics, the inferred rankings perfectly mirrored the discipline’s legendary "Top Five" journals. If a paper was published in one of these venues, its probability of receiving a 4* rating was near 100%. This confirms that in certain fields, journal prestige is an incredibly strong predictor of REF success.
The "Shrinkage Effect": Panels Do Read
Perhaps the most significant finding is what the authors call the "shrinkage effect." The model revealed that top-tier institutions (like Oxford or Cambridge) consistently received higher scores than the model predicted based on their journal profiles alone. Conversely, lower-ranked institutions often performed worse than their journal lists suggested.
This is actually a positive sign for the REF's integrity. It suggests that panels are not just "judging a book by its cover." Instead, they appear to be distinguishing the truly exceptional papers within high-tier journals from the more standard ones, and identifying high-quality work even when it appears in less prestigious venues.
Impact Factor is a Blunt Instrument
The study also highlighted the limitations of generic metrics. In Mathematical Sciences, there was almost zero correlation between a journal’s Impact Factor and its inferred REF quality. This underscores the danger of using one-size-fits-all metrics to judge specialized research fields.
Results and Impact
The model demonstrated high predictive accuracy. The "index of dissimilarity" ($\Delta$)—the proportion of papers that would need to be reclassified to match actual REF results—ranged from a low of 8.2% in Chemistry to 17.9% in Economics.
Furthermore, the authors calculated the "redistribution of monetary reward" ($\Delta_£$). They found that if funding were allocated based solely on journal identity rather than the actual, panel-assessed REF scores, approximately 5.6% to 8.9% of the total research budget would shift between institutions. This suggests that while journal prestige is a major factor, the expert panels' "value-add" significantly alters the final financial outcomes for universities.
Conclusion
While journal prestige is undeniably a massive predictor of a department's success in the REF, it is not the only factor. The "shrinkage effect" provides evidence that expert panels are looking past the journal title to assess the intrinsic quality of the work. However, the high correlation in fields like Economics suggests that the "prestige" of a venue remains a powerful, if not dominant, signal in research assessment.
Summary updated