Semantic Analysis for Automated Evaluation of the Potential Impact of Research Articles

Neslihan Suzen, Alexander N. Gorban, Jeremy Levesley, Evgeny M. MirkesarXiv 2021 · arXiv

Introduction

Predicting the future impact of a research article is a long-standing challenge in scientometrics. Traditionally, impact prediction has relied on extrinsic factors such as author reputation, journal prestige, or bibliography size. However, these metrics often overlook the intrinsic value of the ideas expressed within the text itself. This paper investigates whether the "meaning" of a research abstract, quantified through information theory, can serve as a reliable predictor of citation counts.

Quantifying Meaning: The Informational Text Representation Method

Standard Natural Language Processing (NLP) models, such as Bag-of-Words (BoW) or TF-IDF, often produce sparse data that fails to capture nuanced scientific context. To address this, the authors introduce "informational semantics," a method that maps research abstracts into a high-dimensional "Meaning Space" (MS).

Constructing the Meaning Space

The Meaning Space is defined by 252 dimensions, each corresponding to a Web of Science (WoS) scientific category. Every word in the dictionary is represented as a vector within this space. The value for each dimension is determined by the Relative Information Gain (RIG), which measures how much a word's presence indicates a paper's likelihood of belonging to a specific category.

From Word Clouds to Feature Vectors

An abstract is treated not as a simple sequence of words, but as a statistical "cloud" of word vectors. To facilitate machine learning classification, this cloud is condensed into several feature vectors (FVT):

  • Mean Vector ($\mu$): The average position of all words in the abstract.
  • Principal Components ($PC_1$): The primary direction of variance within the word cloud.
  • K-means Centroids ($c_1, c_2$): The centers of two internal clusters, capturing semantic shifts within the text.

Experimental Setup

The study utilized the Leicester Scientific Corpus (LSC), comprising over 1.6 million abstracts from 2014. Citation data from the Web of Science was tracked over a four-year period (2014–2018). The researchers focused their analysis on three distinct fields: Applied Mathematics, Biology, and Management. Binary classification was performed using Linear Discriminant Analysis (LDA) and k-Nearest Neighbors (kNN) to distinguish between "Highly Cited" and "Little Cited" articles.

Results and Discussion

The informational semantics approach consistently outperformed random chance across all tested categories, particularly when distinguishing between the top and bottom quartiles of citation counts.

Performance by Field

The model demonstrated significant disciplinary sensitivity. It achieved its highest accuracy in Management and Biology. In Management, using LDA on the 252-dimensional space, the model reached 83.22% sensitivity and 81.81% specificity. The authors suggest these fields possess a more homogeneous high-impact vocabulary compared to others.

Conversely, Applied Mathematics proved the most challenging to predict. This is likely due to its multidisciplinary nature; a high-impact math paper's semantic profile varies significantly depending on whether its application is in physics, economics, or biology.

Feature Synergy

The most effective classification results generally came from combining the Mean Vector ($\mu$) with the first Principal Component ($PC_1$). This suggests that both the central topic (the "average" meaning) and the breadth of topics (the variance) are critical indicators of a paper's potential impact.

Conclusion

By treating text as a distribution in a mathematically rigorous "Meaning Space," this research provides a powerful tool for objective impact evaluation. While the current model focuses on binary classification and ignores metadata to isolate semantic power, it establishes a foundation for more complex predictive systems. Future work will explore continuous citation count prediction and the integration of semantic features with traditional bibliometric metadata to further enhance accuracy.

Summary updated

Abstract

Can the analysis of the semantics of words used in the text of a scientific paper predict its future impact measured by citations? This study details examples of automated text classification that achieved 80% success rate in distinguishing between highly-cited and little-cited articles. Automated intelligent systems allow the identification of promising works that could become influential in the scientific community. The problems of quantifying the meaning of texts and representation of human language have been clear since the inception of Natural Language Processing. This paper presents a novel method for vector representation of text meaning based on information theory and show how this informational semantics is used for text classification on the basis of the Leicester Scientific Corpus. We describe the experimental framework used to evaluate the impact of scientific articles through their informational semantics. Our interest is in citation classification to discover how important semantics of texts are in predicting the citation count. We propose the semantics of texts as an important factor for citation prediction. For each article, our system extracts the abstract of paper, represents the words of the abstract as vectors in Meaning Space, automatically analyses the distribution of scientific categories (Web of Science categories) within the text of abstract, and then classifies papers according to citation counts (highly-cited, little-cited). We show that an informational approach to representing the meaning of a text has offered a way to effectively predict the scientific impact of research papers.
BibTeX
@misc{suzen2021semantic,
  author = {Neslihan Suzen and Alexander N. Gorban and Jeremy Levesley and Evgeny M. Mirkes},
  title = {{Semantic Analysis for Automated Evaluation of the Potential Impact of Research Articles}},
  year = {2021},
  eprint = {2104.12869},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.12869},
}
Figure