The CSO Classifier: Ontology-Driven Detection of Research Topics in Scholarly Articles

Angelo Antonio Salatino, Francesco Osborne, Thiviyan Thanapalasingam, Enrico MottaTpdl 2019 · DOI

In the ever-expanding universe of scientific literature, staying on top of research trends is a Herculean task. How do we categorize millions of papers so that researchers can actually find what they need? Traditional methods often fall into two camps: rigid manual classification (which doesn't scale) or "fuzzy" probabilistic models like Latent Dirichlet Allocation (LDA), which are often hard for humans to interpret.

A team from The Open University has introduced a middle ground: the CSO Classifier. This unsupervised tool leverages a massive, granular ontology to automatically detect research topics in scholarly articles with high precision and interpretability.

TL;DR

The CSO Classifier is an unsupervised tool that automatically assigns research topics to Computer Science papers by combining syntactic string matching with semantic word embeddings. By leveraging the massive Computer Science Ontology (CSO), it outperforms traditional topic modeling approaches, achieving a 74.1% F-measure in identifying granular research areas.

The Challenge of Scholarly Classification

Classifying research papers is essential for digital libraries, recommendation engines, and "Science of Science" analytics. However, we face a major gap:

  1. Manual Annotation: High quality, but impossible to scale to the millions of papers published annually.
  2. Topic Modeling (e.g., LDA): Scalable, but produces "noisy" results. LDA might tell you a paper is 20% "social" and 30% "network," but it struggles to map those distributions to a formal, human-readable category like "Community Detection."
  3. Vocabulary Mismatch: Authors use different terms for the same concept (e.g., "virtual machines" vs. "VMs").

The authors aim to bridge this gap by creating a classifier that is both unsupervised (requiring no labeled training data) and ontology-driven, ensuring the output is always a set of formally defined, hierarchical research topics.

System Architecture and Methodology

The CSO Classifier processes a paper’s title, abstract, and keywords through a pipeline that integrates external knowledge sources with natural language processing.

The overall architecture of the CSO Classifier, showing the integration of syntactic and semantic modules with the Computer Science Ontology.

1. The Syntactic Module

This module looks for explicit mentions. It extracts n-grams (unigrams, bigrams, and trigrams) from the text and calculates the Levenshtein similarity between these strings and the 14,000+ topic labels in the CSO. This allows the system to catch variations like plurals or hyphenated terms (e.g., "knowledge-based systems" vs. "knowledge based systems").

2. The Semantic Module

To find topics that are implied but not explicitly named, the classifier uses word embeddings (Word2Vec).

The Semantic Module workflow: identifying similar words via Word2vec and mapping them back to the CSO.
  • Entity Extraction: It uses Part-of-Speech (POS) tagging to identify noun phrases.
  • Similarity Mapping: It looks for the top 10 most similar words in the embedding space for each phrase. If these neighbors match a CSO concept, that concept is flagged.
  • Ranking & Selection: Topics are ranked based on frequency and "diversity" (how many different phrases led to that topic). To decide where to "cut off" the list of suggested topics, the system uses the elbow method, a heuristic used to find the optimal point in a mathematical curve.

3. Enrichment and Post-processing

Finally, the classifier uses the ontology’s hierarchical structure (superTopicOf relations) to include parent categories. If a paper is about "Deep Learning," the classifier automatically infers it is also about "Machine Learning" and "Artificial Intelligence."

Datasets and Evaluation

  • Training Data: The Word2Vec model was trained on a massive corpus of 4.6 million English papers from the Microsoft Academic Graph (MAG).
  • The Ontology: The Computer Science Ontology (CSO), which contains 14k topics and 162k semantic relationships.
  • Evaluation Gold Standard: The authors introduced a new benchmark consisting of 70 papers from the fields of Semantic Web, NLP, and Data Mining, manually annotated by 21 domain experts.

Key Results and Insights

The CSO Classifier was tested against 13 other approaches, including various LDA configurations and the previous tool used by Springer Nature editors (STM).

  • Top Performance: The CSO Classifier (CSO-C) achieved an F1-measure of 74.1%, significantly outperforming the baseline LDA models (which struggled to break 15% F1) and the previous STM tool (67.6% F1).
  • Precision vs. Recall: Purely syntactic methods (like STM) reached high precision (80.8%) but suffered from low recall (58.2%). The CSO Classifier’s semantic module successfully boosted recall to 75.3% without a catastrophic drop in precision.
  • Interpretability over Probability: Unlike LDA, which represents topics as clusters of words, the CSO Classifier points to a specific node in a curated ontology. This makes the output immediately useful for faceted search and library management.

Limitations and Future Work

Currently, the classifier is strictly tuned for Computer Science. Expanding to fields like Medicine or Engineering will require different ontologies (like MeSH) and new word embedding models. The team plans to integrate the classifier into the editorial pipelines of major publishers like Springer Nature to assist editors in categorizing conference proceedings automatically.

Summary updated

BibTeX
@inproceedings{salatino2019cso,
  author = {Angelo Antonio Salatino and Francesco Osborne and Thiviyan Thanapalasingam and Enrico Motta},
  title = {{The CSO Classifier: Ontology-Driven Detection of Research Topics in Scholarly Articles}},
  booktitle = {Tpdl},
  year = {2019},
  doi = {10.1007/978-3-030-30760-8\_26},
}
Figure