Could Bibliometrics Reveal Top Science and Technology Achievements and Researchers? The Case for Evaluatology-based Science and Technology Evaluation

Guoxin Kang, Wanling Gao, Lei Wang, Chunjie Luo, Hainan Ye, Qian He, Shaopeng Dai, Jianfeng ZhanarXiv 2024 · OpenReview

TL;DR

This paper argues that traditional bibliometrics, relying on publication and citation counts, fall short in identifying truly groundbreaking science and technology achievements and researchers. To address this, they propose "evaluatology," a new framework that uses an "extended evaluation condition" and four types of relationships to comprehensively assess scientific progress, as demonstrated by a case study in chip technology.

What Problem Does This Paper Address?

The authors highlight a critical limitation in how we currently evaluate scientific and technological progress: bibliometrics. While widely used, metrics like publication numbers and citation counts are easily manipulated, fail to capture non-publication-based innovations (like the Linux operating system), and often prioritize quantity over quality. This leads to a biased and incomplete picture, potentially overlooking significant contributions, especially in emerging fields or those with unconventional output. The paper aims to bridge this gap by proposing a more robust and nuanced evaluation methodology.

Methods

The core of the proposed solution is "evaluatology," a framework built upon the concept of an "extended evaluation condition" (EC). This EC is a multi-layered structure that breaks down a field into problem domains, sub-problem domains, and so on, down to specific algorithms and support systems.

Key methodological contributions include:

  • Extended Evaluation Condition (EC): A nine-component framework to comprehensively define a field and its components.
  • Four Relationships: The paper defines four ways to connect achievements:
  • Pioneering: Achievements that open entirely new research directions.
  • Progressive: Achievements that build upon and acknowledge prior work (via citations).
  • Parallel: Achievements addressing similar problems within a similar timeframe.
  • Related but Not Connected: Achievements addressing similar problems but without explicit citation links and not occurring simultaneously.
  • Real-world, Perfect, and Pragmatic Evaluation Models: The authors propose a progression from a real-world snapshot of achievements to a "perfect" model that traces evolution, and finally to a "pragmatic" model that filters for top achievements.
  • Top N @X @Y Methodology: A systematic procedure to identify the top N achievements within a specific timeframe (X) and field (Y) by applying the evaluatology framework and a four-round filtering process.

Datasets

The paper primarily uses existing datasets for its case study:

  • Chip100: A dataset developed by BenchCouncil, aiming to identify the top 100 groundbreaking and influential achievements in chip technology from the 1940s to 2023.
  • CSRankings: A database that ranks institutions and researchers based on the number of publications in top-tier computer science conferences.
  • Highly Cited Researchers (Elsevier): A list identifying researchers with a significant number of highly cited papers, representing the top 0.1% of researchers globally.

The paper implicitly uses data from these sources to compare their findings with their proposed evaluatology method.

Key Insights and Ideas

  • Bibliometrics are Insufficient: The paper strongly argues that traditional bibliometric metrics are fundamentally flawed for capturing the full spectrum of scientific and technological impact.
  • Context is King: The "extended evaluation condition" emphasizes the importance of understanding the hierarchical structure of a field and the specific components (problems, algorithms, etc.) that achievements address.
  • Beyond Citations: The four relationship types allow for the recognition of achievements that might not have high citation counts but are nonetheless crucial (e.g., pioneering work, parallel innovations).
  • Holistic Evaluation: Evaluatology aims for a more comprehensive assessment by considering not just publications but also the underlying problem domains, algorithms, and their evolutionary connections.
  • Chip Technology as a Proving Ground: The case study on chip technology demonstrates how this new methodology can uncover significant achievements and researchers that bibliometrics miss.

Key Results

The paper presents a case study comparing its evaluatology-based approach (specifically the Chip100 dataset) with CSRankings and Elsevier's Highly Cited Researchers list. The key finding is that these methods produce vastly different results.

  • Divergent Rankings: Table 2 clearly shows that the top achievements, contributors, and institutions identified by Chip100 are largely distinct from those identified by CSRankings and Highly Cited Researchers. For instance, none of the top five contributors in Chip100 appear in the top five of CSRankings.
  • Identification of Overlooked Achievements: The Chip100 dataset, using the evaluatology framework, successfully identified significant achievements like the Von Neumann Architecture, ISA, and Boolean Algebra, which are crucial to chip technology but might not be as prominently featured or ranked by bibliometric methods alone.
  • Qualitative Insights: The paper provides examples of how the relationships (pioneering, progressive, parallel) help understand the evolution and interconnectedness of achievements within the chip field, offering a richer narrative than simple citation counts.

Limitations and Future Work

  • Subjectivity in "Pioneering": While the paper defines pioneering relationships, the identification of what constitutes a "new research direction" can still involve a degree of subjective interpretation.
  • Data Availability and Granularity: The effectiveness of the evaluatology framework relies on having detailed information about achievements, including their EC components, temporal data, and references. Obtaining this granular data for all achievements across all fields can be challenging.
  • Computational Complexity: Building and analyzing "perfect" and "pragmatic" evaluation models for vast scientific landscapes could be computationally intensive.
  • Future Work: The paper suggests that the evaluatology framework can be applied to other fields beyond chip technology. Further research could focus on refining the weighting of different relationships and components within the EC, as well as developing more automated tools for data collection and analysis.

Summary updated

Abstract

By utilizing statistical methods to analyze bibliographic data, bibliometrics faces inherent limitations in identifying the most significant science and technology achievements and researchers. To overcome this challenge, we present an evaluatology-based science and technology evaluation methodology. At the heart of this approach lies the concept of an extended evaluation condition, encompassing eight crucial components derived from a field. We define four relationships that illustrate the connections among various achievements based on their mapped extended EC components, as well as their temporal and citation links. Within a relationship under an extended evaluation condition, evaluators can effectively compare these achievements by carefully addressing the influence of confounding variables. We establish a real-world evaluation system encompassing an entire collection of achievements, each of which is mapped to several components of an extended EC. Within a specific field like chip technology or open source, we construct a perfect evaluation model that can accurately trace the evolution and development of all achievements in terms of four relationships based on the real-world evaluation system. Building upon the foundation of the perfect evaluation model, we put forth four-round rules to eliminate non-significant achievements by utilizing four relationships. This process allows us to establish a pragmatic evaluation model that effectively captures the essential achievements, serving as a curated collection of the top N achievements within a specific field during a specific timeframe. We present a case study on the top 100 Chip achievements which highlights its practical application and efficacy in identifying significant achievements and researchers that otherwise can not be identified by using bibliometrics.
BibTeX
@misc{kang2024could,
  author = {Guoxin Kang and Wanling Gao and Lei Wang and Chunjie Luo and Hainan Ye and Qian He and Shaopeng Dai and Jianfeng Zhan},
  title = {{Could Bibliometrics Reveal Top Science and Technology Achievements and Researchers? The Case for Evaluatology-based Science and Technology Evaluation}},
  year = {2024},
  eprint = {2408.12158},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.12158},
}
Figure