U-Index, a dataset and an impact metric for informatics tools and databases

Alison Callahan, Rainer Winnenburg, Nigam H. ShahScientific 2018 · DOI

TL;DR

This paper introduces the "u-Index," a novel metric to quantify the impact of informatics tools and databases by distinguishing between "usage" and "awareness" citations. They've also released a comprehensive dataset of informatics resource publications and their citation network, enabling researchers to better assess the real-world adoption and influence of these critical resources.

What Problem Does This Paper Address?

In the rapidly evolving field of informatics, software tools and databases are fundamental to biomedical research. However, quantifying their true impact, value, and return on investment has been a significant challenge. While citations are a common proxy for impact, simply counting them doesn't tell the whole story. A citation in a methods section often signifies actual usage, whereas a citation in an introduction or discussion might just indicate awareness of the resource's existence. This ambiguity makes it difficult to discern which tools are genuinely being adopted and used by the research community versus those that are merely known. Existing methods for tracking resource usage, like registries, can be labor-intensive and quickly become outdated.

Methods

The core of this work lies in developing a nuanced approach to citation analysis. The authors created a "citing universe" by processing the PubMed Central (PMC) Open Access subset, focusing on research articles that contain a "Materials/Methods" section. This section is crucial because citations within it are more likely to represent actual usage of a tool or database.

They then developed the u-Index metric, calculated as:

$$ \text{u-Index} = \frac{\text{total \# citations} \times \text{usage ratio}}{\text{# years since first publication}} $$

where:

  • Total # citations: The overall number of citations a resource receives.
  • Usage ratio: The ratio of "usage citations" (found in the Materials/Methods section) to "awareness citations" (found in other sections).
  • # years since first publication: A normalization factor to account for the age of the resource.

This formula prioritizes resources that are not only frequently cited but also demonstrably used, while also considering their longevity.

Datasets

The paper introduces a new, publicly available dataset of informatics resource publications and their citation network. This dataset was constructed by:

  • Querying PubMed: Using a comprehensive set of MeSH terms and keywords to identify publications describing informatics tools and databases.
  • Processing PubMed Central (PMC) Open Access subset: To build the "citing universe" and extract citation context (usage vs. awareness).
  • Aggregating publications: Grouping multiple publications referring to the same tool or database into a single "resource key."

This dataset contains pre-calculated u-Index values for nearly 100,000 informatics resources, covering citations from 2000 to 2015.

Key Insights and Ideas

  • Usage vs. Awareness is Key: The fundamental insight is that not all citations are equal. Distinguishing between citations that indicate direct use (Methods section) and those that merely signal knowledge (other sections) provides a much more accurate measure of a resource's impact.
  • Context Matters: By leveraging the structured nature of scientific publications (specifically the Materials/Methods section), the authors can infer the purpose of a citation, moving beyond simple citation counts.
  • Age Normalization is Crucial: Older resources naturally accumulate more citations. Normalizing by the number of years since publication prevents established, but perhaps less innovative, tools from unfairly dominating impact metrics.
  • Domain-Specific Usage Patterns: The study highlights that some tools are used more in certain domains. For instance, tools like BLAST and CLUSTAL are heavily used in biology and medicine, while in bioinformatics, they might be cited more for comparative purposes (awareness) than direct application.

Key Results

  • The authors successfully curated a dataset of approximately 100,000 unique informatics resources.
  • They pre-calculated the u-Index for these resources, providing a ready-to-use impact metric.
  • Their analysis of usage citation proportion across domains (Figure 1) shows that tools like BLAST and CLUSTAL have a higher proportion of usage citations in biology and medicine compared to bioinformatics.
  • The u-Index over time (Figure 2) demonstrates how the impact of tools like Muscle, Clustal, and MAFFT has grown significantly, while text mining tools, though generally having lower u-Index values, are also showing increasing impact, with LINNAEUS being a notable example.

Limitations and Future Work

  • Citation Bias: The authors acknowledge that citation practices can be biased towards well-known publications, and not all resources are cited even when used.
  • Under-reporting: The method relies on structured citations, which may miss instances where a resource is mentioned but not formally cited, or cited in a non-standard way.
  • Negative Citations: While not observed in their random subset analysis, the current u-Index doesn't distinguish between positive and negative sentiment in citations.
  • Citing Universe Specificity: The u-Index is sensitive to the chosen citing universe. While they used PMC Open Access for reproducibility, other universes could yield different results.

Future work could involve incorporating data from other sources (e.g., social media, news) to capture a broader range of mentions, developing more sophisticated methods to disambiguate resource names, and exploring the impact of negative citations. The authors also emphasize the public availability of their dataset and code, inviting others to build upon their work and devise novel analysis regimes.

Summary updated

Abstract

Measuring the usage of informatics resources such as software tools and databases is essential to quantifying their impact, value and return on investment. We have developed a publicly available dataset of informatics resource publications and their citation network, along with an associated metric (u-Index) to measure informatics resources' impact over time. Our dataset differentiates the context in which citations occur to distinguish between 'awareness' and 'usage', and uses a citing universe of open access publications to derive citation counts for quantifying impact. Resources with a high ratio of usage citations to awareness citations are likely to be widely used by others and have a high u-Index score. We have pre-calculated the u-Index for nearly 100,000 informatics resources. We demonstrate how the u-Index can be used to track informatics resource impact over time. The method of calculating the u-Index metric, the pre-computed u-Index values, and the dataset we compiled to calculate the u-Index are publicly available.
BibTeX
@inproceedings{callahan2018u,
  author = {Alison Callahan and Rainer Winnenburg and Nigam H. Shah},
  title = {{U-Index, a dataset and an impact metric for informatics tools and databases}},
  booktitle = {Scientific},
  year = {2018},
  doi = {10.1038/sdata.2018.43},
}
Figure