A Contrast Pattern-Based Scientometric Study of the QS World University Ranking

Octavio Loyola-González, Miguel Angel Medina-Pérez, Raymundo Adrián Coronilla Valdez, Kim-Kwang Raymond ChooIEEE Access 2020 · DOI · OpenReview

Introduction

Global university rankings, such as the QS World University Ranking, serve as critical benchmarks that influence institutional prestige, student enrollment, and funding allocations. Despite their significance, these rankings are frequently criticized for their lack of transparency, often being perceived as "black boxes" driven by subjective surveys. While existing research has explored correlations between different ranking systems, there remains a lack of actionable, data-driven insights that university administrators can use to inform strategic planning. This study addresses this gap by applying explainable AI (XAI) to identify the specific research output patterns that distinguish elite institutions from their peers.

Methodology

The researchers employed Contrast Pattern Mining to move beyond traditional "black-box" machine learning models toward more interpretable results. The methodology followed a three-step process:

  1. Benchmarking Separability: The team evaluated 18 state-of-the-art classifiers—including Random Forests, Support Vector Machines (SVMs), and Neural Networks—to confirm that the Top 100 and the 101–200 tiers are statistically distinct based solely on research metrics.
  2. Pattern Extraction: Using the Hellinger Distance-based Random Forest (HRFm) algorithm, the study extracted "contrast patterns." These are conjunctions of relational statements (e.g., [Citations > X] AND [Authors > Y]) that occur significantly more frequently in the Top 100 class than in the 101–200 class.
  3. Visualization: To facilitate decision-making, the Bokeh library was used to generate multi-dimensional scatter plots, allowing stakeholders to visualize how specific institutions align with these elite patterns.

Data and Features

The study utilized a custom dataset merging the 2020 QS World University Rankings (Top 200) with comprehensive scientometric data from SciVal (Scopus).

  • Scope: 200 universities, divided into "Top 100" and "101–200" classes.
  • Features: 34 distinct metrics covering the period from 2016 to 2018, including total publications, citation counts, h5-index, citations per author, and international collaboration percentages.

Key Insights and Results

The analysis revealed that the Top 100 tier is mathematically separable from the 101–200 tier with an average accuracy of 71% based on research output alone. The PBC4cip (Pattern-Based Classifier) achieved a peak accuracy of 73.5%, demonstrating the effectiveness of contrast patterns for socio-economic data.

The Primacy of the h5-Index

The h5Index (the h-index over the most recent five years) emerged as the most critical feature for elite status. Its frequent appearance in high-quality contrast patterns suggests that sustained, high-impact research output is the primary gatekeeper to the global Top 100.

Volume vs. Collaboration

While international collaboration is a widely promoted metric, the study found that absolute volume—specifically total citations and the number of publishing authors—had a stronger contrast power in defining the elite tier. Top 100 universities typically publish over 7,400 papers and receive more than 35,000 citations annually.

Actionable Strategic Thresholds

The study identified specific "tipping points" for institutional strategy. For instance, one pattern (CP7) indicated a 50% likelihood of Top 100 membership for universities with more than 75,225 citations in 2017 and over 6,347 publishing authors in 2016.

Conclusion and Future Work

This research provides a roadmap for universities seeking to improve their global standing by identifying the specific thresholds of research excellence required for elite status. However, the study is limited to the Top 200 institutions and the specific criteria of the QS ranking. Future research will expand this methodology to other major ranking systems, such as the Academic Ranking of World Universities (ARWU) and the Times Higher Education (THE) rankings, to determine if these "patterns of excellence" are universal across different evaluative frameworks.

Summary updated

Abstract

Despite the shortcomings and criticisms of world university rankings, such metrics are widely used by students and parents to select institutions and by educational institutions to attract talented students and researchers, as well as funding. This article introduces the first contrast pattern-based scientometric study of world university rankings. Specifically, this study collects a database containing 34 features, which describe the essential research indicators for the top 200 universities in the Quacquarelli Symonds (QS) ranking. The use of 18 state-of-the-art classifiers in this database shows that the top 100 universities in the QS World University Rankings are separable from the remaining compared universities, achieving an average accuracy of 71%. Additionally, using a contrast pattern mining algorithm, a set of patterns describing the top 100 universities is extracted based on scientometric features. Additionally, this study proposes an approach for visualizing the extracted patterns to facilitate the decision-makers, such as senior university managers, in formulating and evaluating their research (ranking) strategies.
BibTeX
@article{loyolagonzalez2020contrast,
  author = {Octavio Loyola-González and Miguel Angel Medina-Pérez and Raymundo Adrián Coronilla Valdez and Kim-Kwang Raymond Choo},
  title = {{A Contrast Pattern-Based Scientometric Study of the QS World University Ranking}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3037665},
}
Figure