ACE 2.0: A Comprehensive tool for automatic extraction, analysis, and digital profiling of the researchers in Scientific Communities

Syed Tahseen Raza Rizvi, Sheraz Ahmed, Andreas DengelSoc. Netw. Anal. Min 2023 · DOI

Introduction

In the current academic landscape, the sheer volume of scientific output has made manual oversight of research communities nearly impossible. Traditional evaluation methods often rely on "shallow" metrics like the h-index or raw citation counts. However, these metrics fail to distinguish between positive and negative citations, are susceptible to manipulation via self-citation, and often disadvantage emerging researchers. ACE 2.0 (Academic Community Explorer) addresses these limitations by providing an end-to-end automated system for the digital profiling of researchers, moving beyond simple counts toward a nuanced "Semantic Index."

System Architecture

ACE 2.0 is designed as a modular pipeline that transforms raw input documents into a comprehensive visualization of a scientific community's health and evolution. The system integrates state-of-the-art models for extraction, sentiment analysis, and graph-based profiling.

The ACE 2.0 modular pipeline, from input documents to the final visualization engine.

Core Modules

Bibliographic Reference Extraction (DeepBiRD)

A critical challenge in profiling is the accurate extraction of references from diverse document styles. ACE 2.0 utilizes DeepBiRD, which adopts a layout-based approach rather than traditional text parsing. By treating documents as images, it employs a Mask-RCNN architecture to detect references based on visual cues such as indentation and line spacing.

DeepBiRD architecture utilizing Mask-RCNN for layout-based reference detection.

Sentiment Analysis (ImpactCite)

To provide a qualitative assessment of impact, the ImpactCite module uses an XLNet-based model to classify citation intent. It categorizes citations as Positive, Negative, or Neutral, allowing the system to filter out critiques and focus on constructive influence.

Keyword and Topic Modeling (CoCoNoW)

The CoCoNoW module extracts keywords by building a "Connectivity-Aware Graph." These keywords are mapped to the Computer Science Ontology (CSO), enabling the system to track the dynamic evolution of research topics over time and identify emerging trends.

The Semantic Index

The core innovation of ACE 2.0 is the Semantic Index. Unlike the h-index, this metric calculates a researcher's score by multiplying the number of positive, non-self-citations by a composite of five graph centrality measures (Degree, Eigenvector, Betweenness, Closeness, and Indegree). This approach allows for the identification of specific community roles, such as "Opinion Leaders" or "Community Connectors."

Evaluation and Results

The individual components of ACE 2.0 demonstrated high performance across established benchmarks:

  • DeepBiRD: Achieved state-of-the-art results with an AP50 of 97.59% on the BibX dataset and 98.56% on BibLy.
  • ImpactCite: Outperformed standard BERT and ALBERT models with a Macro-f1 score of 77.73 in sentiment classification.
  • CoCoNoW: Consistently surpassed traditional methods like TF-IDF and TextRank in keyword extraction accuracy across multiple datasets (Hulth2003, SemEval2010, and NLM500).

Discussion and Conclusion

ACE 2.0 represents a significant step toward more transparent and content-oriented research assessment, aligning with the goals of the Declaration on Research Assessment (DORA). By leveraging layout-based extraction and graph theory, the system provides a fairer evaluation of researcher impact.

Future work will focus on improving author disambiguation—a persistent challenge in automated profiling—and scaling the system beyond the Computer Science domain to provide a global view of scientific progress. Currently, the system offers a powerful tool for understanding the internal dynamics and evolutionary trends of specific scientific communities.

Summary updated

BibTeX
@article{rizvi2023ace,
  author = {Syed Tahseen Raza Rizvi and Sheraz Ahmed and Andreas Dengel},
  title = {{ACE 2.0: A Comprehensive tool for automatic extraction, analysis, and digital profiling of the researchers in Scientific Communities}},
  journal = {Soc. Netw. Anal. Min.},
  year = {2023},
  doi = {10.1007/s13278-023-01085-w},
}
Figure