Correlation and Interaction Visualization of Altmetric Indicators Extracted From Scholarly Social Network Activities: Dimensions and Structure

Chunli Liu, Yue Xu, Hui Wu, Si Si Chen, Ji GuoJournal of Medical Internet Research 2013 · DOI

Abstract

In the Web 2.0 era, traditional scholarly metrics like citation counts and journal impact factors are no longer sufficient to capture the immediate and multidimensional nature of scientific communication. This study investigates the dimensions and interaction structures of altmetric indicators using a robust dataset of 33,128 academic articles from the Public Library of Science (PLOS). Through Spearman rank correlation, multidimensional scaling (MDS), and social network analysis (SNA), we identified three distinct clusters of metrics: traditional, active altmetrics, and inactive altmetrics. Our findings suggest a "transfer phenomenon" where active altmetrics, such as blog posts and comment threads, serve as intermediate stations that accelerate an article's social impact toward formal academic citations.

Introduction: Beyond the Impact Factor

For decades, the "Impact Factor" and citation counts have served as the gold standards for measuring scientific importance. However, these metrics are inherently slow, often taking years to reflect the true reach of a paper. In the modern "Web 2.0" landscape, scholars increasingly use social media, blogs, and bookmarking tools to share and discuss research.

These "altmetrics" offer a more immediate, open, and individualized measure of impact. Yet, until recently, the internal structure of these indicators—how they relate to each other and to traditional citations—remained a "black box." This research seeks to reveal the underlying dimensions of altmetrics and the pathways through which they interact to form a cohesive picture of scholarly influence.

Methods and Data

The researchers analyzed a massive dataset of article-level metrics (ALM) from the PLOS API, covering 33,128 articles. They tracked 20 distinct indicators, ranging from traditional citations (CrossRef, Scopus) and usage data (HTML views, PDF downloads) to social signals (tweets, blog posts, and Mendeley bookmarks).

Because altmetric data is typically non-normal and positively skewed, the team employed non-parametric statistical methods:

  • Spearman Rank Correlation: Used to measure the strength and direction of relationships between the 20 variables.
  • Multidimensional Scaling (MDS): A visualization technique that maps the correlation matrix into a two-dimensional space, allowing researchers to see which metrics naturally cluster together.
  • Social Network Analysis (SNA): Using software like NetDraw, the metrics were treated as nodes in a network to identify "hubs" and "bridges" that connect different types of impact.

Key Insights

The Pareto Law of Scholarly Attention

The study found that downloads and page views follow the Pareto law, commonly known as the 80/20 rule. Approximately 80% of the total online "buzz" is generated by only 20% of the articles. This suggests that while altmetrics provide a broader view, scholarly attention remains highly concentrated on a subset of high-impact work.

Three Dimensions of Impact

The MDS analysis revealed that the 20 indicators fall into three distinct functional clusters:

  1. Traditional Metrics: This cluster includes formal citations and, interestingly, usage data like HTML views and PDF downloads. The extremely high correlation between views and downloads (rho = .899) suggests these are essentially two sides of the same "usage" coin, which serves as a precursor to formal citation.
  2. Active Altmetrics: This group includes trackbacks, ratings, and comment threads. These represent deep, active engagement where the audience is interacting with the content rather than just consuming it.
  3. Inactive/Niche Altmetrics: These are less frequent indicators, such as specific social bookmarking services (e.g., CiteULike), which provide unique but less dense signals of impact.

The Impact Transfer Phenomenon

The most compelling discovery is the "transfer phenomenon." Social network diagrams identified specific indicators—notably blog posts (ResearchBlogging.org) and comment/note threads—that act as "transfer stations." These metrics have the highest number of ties to other groups, functioning as the connective tissue that transports an article's social momentum into the realm of formal academic citations.

Conclusion

Altmetrics are not merely a replacement for traditional metrics but a vital complement to them. They are superior to traditional filters for assessing scholarly impact across multiple dimensions and social structures. By understanding the transfer paths from active social engagement to formal citations, academic institutions can better evaluate the multifaceted impact of research in the digital age. Future evaluations should consider not only traditional citations but also "active altmetrics" to capture the full spectrum of a researcher's contribution.

Summary updated

Abstract

Background: Citation counts for peer-reviewed articles and the impact factor of journals have long been indicators of article importance or quality. In the Web 2.0 era, growing numbers of scholars are using scholarly social network tools to communicate scientific ideas with colleagues, thereby making traditional indicators less sufficient, immediate, and comprehensive. In these new situations, the altmetric indicators offer alternative measures that reflect the multidimensional nature of scholarly impact in an immediate, open, and individualized way. In this direction of research, some studies have demonstrated the correlation between altmetrics and traditional metrics with different samples. However, up to now, there has been relatively little research done on the dimension and interaction structure of altmetrics. Objective: Our goal was to reveal the number of dimensions that altmetric indicators should be divided into and the structure in which altmetric indicators interact with each other. Methods: Because an article-level metrics dataset is collected from scholarly social media and open access platforms, it is one of the most robust samples available to study altmetric indicators. Therefore, we downloaded a large dataset containing activity data in 20 types of metrics present in 33,128 academic articles from the application programming interface website. First, we analyzed the correlation among altmetric indicators using Spearman rank correlation. Second, we visualized the multiple correlation coefficient matrixes with graduated colors. Third, inputting the correlation matrix, we drew an MDS diagram to demonstrate the dimension for altmetric indicators. For correlation structure, we used a social network map to represent the social relationships and the strength of relations. Results: We found that the distribution of altmetric indicators is significantly non-normal and positively skewed. The distribution of downloads and page views follows the Pareto law. Moreover, we found that the Spearman coefficients from 91.58% of the pairs of variables indicate statistical significance at the .01 level. The non-metric MDS map divided the 20 altmetric indicators into three clusters: traditional metrics, active altmetrics, and inactive altmetrics. The social network diagram showed two subgroups that are tied to each other but not to other groups, thus indicating an intersection between altmetrics and traditional metric indicators. Conclusions: Altmetrics complement, and most correlate significantly with, traditional measures. Therefore, in future evaluations of the social impact of articles, we should consider not only traditional metrics but also active altmetrics. There may also be a transfer phenomenon for the social impact of academic articles. The impact transfer path has transfer, or intermediate, stations that transport and accelerate article social impact from active altmetrics to traditional metrics and vice versa. This discovery will be helpful to explain the impact transfer mechanism of articles in the Web 2.0 era. Hence, altmetrics are in fact superior to traditional filters for assessing scholarly impact in multiple dimensions and in terms of social structure.
BibTeX
@article{liu2013correlation,
  author = {Chunli Liu and Yue Xu and Hui Wu and Si Si Chen and Ji Guo},
  title = {{Correlation and Interaction Visualization of Altmetric Indicators Extracted From Scholarly Social Network Activities: Dimensions and Structure}},
  journal = {Journal of Medical Internet Research},
  year = {2013},
  doi = {10.2196/jmir.2707},
}
Figure