An article level metric in the context of research community

Yu Liu, Zhen Huang, Jing Fang, Yizhou YanWWW 2014 · DOI

TL;DR

This paper introduces ALM_RC, a novel article-level metric that assesses research paper impact by considering researchers, journals, social bookmarks, and citations within a specific research community. It aims to provide a more comprehensive and context-aware evaluation than traditional metrics, especially for newly published work.

What Problem Does This Paper Address?

The sheer volume of research papers published today makes it challenging for researchers to identify the most relevant and impactful work. Traditional metrics like the Journal Impact Factor (JIF) have limitations: JIF averages citations across an entire journal, not assessing individual papers, and citation counts suffer from a significant time-lag, making them less effective for recent publications. While Web 2.0 tools have introduced new metrics (e.g., social bookmarks, blog coverage), a systematic way to integrate these diverse factors for a holistic assessment is still needed. The authors argue that research communities, being stable environments focused on specific directions, offer a valuable context for evaluating paper impact.

Methods

The core of the paper is the proposed Article Level Metric in a Research Community (ALM_RC). This metric integrates four key factors:

  • Researcher Impact (A1): Assessed using the R-index, a variant of the H-index that is more sensitive to the number of citations of highly cited papers. If an article has multiple authors, the highest R-index among them is used.
  • Journal Impact (A2): Also evaluated using the R-index, applied to journals within the research community.
  • Social Bookmark Impact (A3): Measures the popularity of an article based on social bookmarking data, specifically from Mendeley.
  • Citation Impact (A4): The traditional measure of citations, sourced from Google Scholar.

The ALM_RC formula is a weighted sum of these factors: ALM_RC = w1*A1 + w2*A2 + w3*A3 + A4. The weights (w1, w2, w3) are determined by a novel method: they are the reciprocals of the "critical impact value" of the top 20% of papers in the respective impact lists (researchers, journals, social bookmarks). This "critical impact value" is derived using the 80/20 rule, suggesting that a small percentage of causes drive a large percentage of effects. For social bookmarks, the critical impact is specifically calculated for the most recent year.

Datasets

The authors evaluated their metric using a research community focused on the H-index. They collected data for 820 articles from the bibliography at http://sci2s.ugr.es/hindex/biblio. Citation data was obtained from Google Scholar, and social bookmark data from Mendeley, both as of October 2012. The paper does not introduce a new benchmark dataset but uses existing data from a specific research field.

Key Insights and Ideas

  • Context Matters: Evaluating papers within a research community provides a more stable and relevant assessment than a universal metric.
  • Complementary Metrics: Different factors (researchers, journals, social bookmarks, citations) have different time-dependent impacts. Researchers, journals, and social bookmarks are more influential for recently published papers, while citations become dominant over time.
  • R-index Superiority: The R-index is highlighted as a better measure for researcher and journal impact compared to the traditional H-index, especially for those with fewer but highly cited papers.
  • Weighting Scheme: The method for determining weights based on the "critical impact value" of the top 20% is a novel approach to balance the influence of different factors.
  • Robustness: The proposed ALM_RC metric demonstrates robustness, meaning that small variations in the weights do not significantly alter the ranking of papers.

Key Results

  • The ALM_RC metric was calculated for papers in the H-index research community.
  • Time-dependent influence: Analysis of the percentage contribution of each factor (Table 5) showed that for recently published papers (e.g., 2005-2009), citations (A4) had a lower percentage impact compared to researchers (A1), journals (A2), and social bookmarks (A3). As papers aged, the influence of citations increased significantly. This validates the metric's ability to assess papers across different publication dates.
  • Weight values: The calculated weights for the H-index community were w1=0.25, w2=0.25, and w3=0.14.
  • Sensitivity Analysis: When weights were varied within a reasonable range, the ranking of papers by ALM_RC remained largely consistent, indicating the metric's robustness.
  • The paper provides a list of the top 10 papers published in 2011 according to the ALM_RC metric, intended to guide researchers.

Limitations and Future Work

  • Data Availability: The accuracy of social bookmark data depends on the adoption and availability of services like Mendeley.
  • Research Community Definition: The paper assumes a well-defined research community with a curated list of articles. Defining these communities and ensuring their comprehensiveness can be challenging.
  • Future Work: The authors suggest that more researchers should participate in maintaining article lists for different research fields to leverage ALM_RC effectively. They also imply that further refinement of the weighting scheme and exploration of other Web 2.0 metrics could enhance the metric.

Summary updated

BibTeX
@inproceedings{liu2014article,
  author = {Yu Liu and Zhen Huang and Jing Fang and Yizhou Yan},
  title = {{An article level metric in the context of research community}},
  booktitle = {The Web Conference},
  year = {2014},
  doi = {10.1145/2567948.2579030},
}
Figure