Implications of Web of Science journal impact factor for scientific output evaluation in 16 institutions and investigators' opinion.

Yì Wáng, Richa Arora, Yongdoo Choi, Hsiao‐Wen Chung, V. Egorov, Jens Frahm, Hiroyuki Kudo, Suleyman Kuyumcu, Sophie Laurent, Romaric Loffroy, Simone Maurea, Sameh K. Morcos, Yicheng Ni, Edwin H. G. Oei, Akmal Sabarudin, Xin YuPubMed 2014 · DOI

TL;DR

This paper surveys researchers from 16 institutions across the globe to understand how the Web of Science (WoS) journal impact factor (IF) and citations are used to evaluate scientific output. The findings reveal that IF and ISI citations heavily influence academic life, despite widespread acknowledgment of their limitations, while Google Scholar metrics, though convenient, lack official recognition.

What Problem Does This Paper Address?

The academic world relies heavily on metrics to evaluate research quality and individual researcher performance. However, the suitability and fairness of these metrics are constantly debated. Specifically, the journal impact factor (IF), a measure of the average number of citations received by articles published in a journal, has become a dominant tool for assessing scientific output. Yet, critics argue that IF is an imperfect proxy for individual article quality, can be manipulated, and may not accurately reflect contributions across diverse fields. This paper aims to investigate how this influential metric is actually being implemented and perceived in various institutions worldwide, and to gauge opinions on its overall benefit or harm.

Methods

The study employed a survey-based approach. Sixteen contributors from institutions in Hong Kong SAR, India, Korea, Taiwan, Russia, Germany, Japan, Turkey, Belgium, France, Italy, UK, The Netherlands, Malaysia, and USA were invited to answer six specific questions. These questions explored:

  1. The extent to which WoS journal IF and ISI citation are the primary tools for academic output evaluation in their institutions and countries.
  2. The perceived value of Google citation counts.
  3. How papers published in non-SCI journals (but indexed in PubMed and searchable by Google Scholar) are valued compared to papers in SCI journals with IF.
  4. The personal value placed on publishing in non-SCI versus SCI journals.
  5. General views on metric measurement of scientific output.
  6. An overall assessment of whether WoS journal IF is beneficial or harmful.

The responses were then compiled and presented, with the authors explicitly stating their intention was not to draw conclusions or make recommendations, but rather to present the current status and individual opinions.

Datasets

Not discussed in the paper. The paper relies on survey responses from individual researchers rather than specific datasets for analysis.

Key Insights and Ideas

  • IF Dominance, Despite Doubts: The overwhelming consensus is that WoS journal IF and ISI citations are indeed the primary, or at least core, tools for evaluating academic output in most institutions surveyed. This holds true even when researchers acknowledge the metric's flaws.
  • Google Scholar's Unofficial Role: While convenient and speedy, Google Scholar metrics (like citation counts) have not gained widespread "official" recognition for formal evaluation. They are often used informally by individuals for personal tracking or CV building.
  • The SCI vs. Non-SCI Divide: Papers published in SCI journals with a good IF are generally valued more than those in non-SCI journals, even if the latter are indexed in PubMed and searchable. This creates a significant disadvantage for research published outside the most prestigious journals.
  • Pressure on Junior Faculty: The emphasis on IF places considerable stress on junior faculty members, pushing them towards "hot" research areas that are likely to yield high-IF publications, potentially at the expense of their traditional strengths or less glamorous but important research.
  • Field-Specific Challenges: The IF's inability to be easily compared across disciplines is a significant issue, particularly for fields like radiology, where traditional journals may have lower IFs than those in other clinical specialties, potentially disadvantaging researchers in grant applications or cross-disciplinary collaborations.
  • "Harm" vs. "Benefit": While many acknowledge the IF's utility in providing a quantitative indicator, a significant portion of respondents believe its harms outweigh its benefits, citing its potential to mislead, encourage "number pursuing" over scientific content, and contribute to the rise of predatory journals seeking profit.

Key Results

The survey results indicate that:

  • WoS journal IF and ISI citation are heavily affecting academic life in most institutions.
  • Google citation and evaluation, while convenient, have not gained wide "official" recognition as a tool for scientific output evaluation.
  • Papers published in non-SCI journals are generally valued less than those in SCI journals with an IF, even if the former are indexed in PubMed.
  • Many researchers feel that the IF is more harmful than beneficial, or at least has significant limitations.

Limitations and Future Work

The authors explicitly state a key limitation: only one contributor was invited from each institution. Therefore, their views cannot be confirmed as the consensus representing the official positions of their respective institutions or regions.

Future work could involve expanding the survey to include more contributors from each institution to gain a more representative view. Further research could also explore the development of more comprehensive and equitable metrics for scientific output evaluation that account for cross-disciplinary comparisons and individual article quality beyond journal prestige. The paper also implicitly suggests the need for clearer guidelines and policies to mitigate the negative impacts of IF-driven evaluation.

Summary updated

Abstract

Journal based metrics is known not to be ideal for the measurement of the quality of individual researcher's scientific output. In the current report 16 contributors from Hong Kong SAR, India, Korea, Taiwan, Russia, Germany, Japan, Turkey, Belgium, France, Italy, UK, The Netherlands, Malaysia, and USA are invited. The following six questions were asked: (I) is Web of Sciences journal impact factor (IF) and Institute for Scientific Information (ISI) citation the main academic output performance evaluation tool in your institution? and your country? (II) How does Google citation count in your institution? and your country? (III) If paper is published in a non-SCI journal but it is included in PubMed and searchable by Google scholar, how it is valued when compared with a paper published in a journal with an IF? (IV) Do you value to publish a piece of your work in a non-SCI journal as much as a paper published in a journal with an IF? (V) What is your personal view on the metric measurement of scientific output? (VI) Overall, do you think Web of Sciences journal IF is beneficial, or actually it is doing more harm? The results show that IF and ISI citation is heavily affecting the academic life in most of the institutions. Google citation and evaluation, while is being used and convenient and speedy, has not gain wide 'official' recognition as a tool for scientific output evaluation.
BibTeX
@inproceedings{wang2014implications,
  author = {Yì Wáng and Richa Arora and Yongdoo Choi and Hsiao‐Wen Chung and V. Egorov and Jens Frahm and Hiroyuki Kudo and Suleyman Kuyumcu and Sophie Laurent and Romaric Loffroy and Simone Maurea and Sameh K. Morcos and Yicheng Ni and Edwin H. G. Oei and Akmal Sabarudin and Xin Yu},
  title = {{Implications of Web of Science journal impact factor for scientific output evaluation in 16 institutions and investigators' opinion.}},
  booktitle = {PubMed},
  year = {2014},
  doi = {10.3978/j.issn.2223-4292.2014.11.16},
}
Figure