Artificial Intuition: Efficient Classification of Scientific Abstracts

Harsh Sakhrani, Naseela Pervez, Anirudh Ravikumar, Fred Morstatter, Alexandra Graddy-Reed, Andrea BelzACL 2024

TL;DR

This paper introduces "Artificial Intuition," a novel method for automatically classifying scientific abstracts into coarse domain-specific categories. By leveraging Large Language Models (LLMs) to augment keywords with context-specific metadata, the approach aims to mimic human expert intuition for efficient and accurate classification, overcoming limitations of existing methods.

What Problem Does This Paper Address?

Classifying scientific abstracts is crucial for managing research portfolios, tracking trends, and screening literature. However, automating this task is challenging due to the brevity and dense, context-rich nature of abstracts. Existing methods struggle with:

  • Author self-labeling: Prone to idiosyncratic decisions and inconsistent terminology across disciplines.
  • External ontologies: Often too fine-grained or lack labels for emerging fields.
  • Automated methods: Supervised learning requires extensive manual annotation, which is costly and time-consuming. While unsupervised methods exist, they often fail to capture the nuanced semantic value of key phrases that may appear infrequently.

The paper aims to bridge this gap by developing a unified, coarse-grained taxonomy and an unsupervised methodology that circumvents manual annotation while effectively handling the unique characteristics of scientific abstracts.

Methods

The core of the "Artificial Intuition" method involves a two-stage process:

  1. Label Space Generation:
    • Keyword Extraction: The YAKE! algorithm is used to extract keywords from abstracts.
  2. Metadata Augmentation: A Large Language Model (Mistral-7B) is employed to generate context-specific metadata for these keywords. This is a key innovation, as it provides richer, domain-aware information than generic keyword definitions.
  3. Embedding: The combined keyword-metadata is then processed by a Sentence Transformer to create embeddings.
  4. Clustering: K-means clustering is applied to these embeddings to form coarse-grained labels, with cluster centroids representing the labels.
  5. Label Assignment (Prediction):
    • Coverage Matrix: A coverage matrix is constructed by projecting keyword embeddings onto the label space embeddings. This matrix quantifies how well each label describes the keywords within a document.
  6. Thresholding: A threshold (T) is applied to the coverage matrix to identify significant overlaps between keywords and labels.
  7. Prediction: Documents are assigned labels based on these significant overlaps.

The method is designed to be flexible, allowing for adaptation to new label spaces without retraining.

Datasets

The primary dataset used is a corpus of 1,230 abstracts from the National Aeronautics and Space Administration (NASA) Small Business Innovation Research (SBIR) Program, collected from 2010 to 2015. A subset of 100 abstracts was randomly selected for manual classification to serve as a ground truth for evaluation. No new benchmark datasets were introduced; instead, the authors adapted an existing corpus to their specific classification task.

Key Insights and Ideas

  • LLMs as "Artificial Intuition": The paper posits that LLMs can effectively mimic human expert intuition by providing context-specific metadata, which is crucial for understanding the semantic weight of keywords in scientific abstracts.
  • Context-Specific Metadata is Key: Augmenting keywords with LLM-generated definitions tailored to the specific document's context is more effective than using generic definitions.
  • Label Space Generation is a Clustering Problem: Framing label space creation as a clustering problem allows for the discovery of natural thematic groupings within a document corpus.
  • Coverage and Redundancy as Evaluation Metrics: The introduction of "coverage" (how well labels span the document space) and "redundancy" (orthogonality of label embeddings) provides novel ways to evaluate the quality of a generated label space.
  • Trade-off Between Coverage and Redundancy: The study highlights an inherent trade-off: increasing the number of labels (k) generally improves coverage but can also lead to increased redundancy.

Key Results

  • The proposed method achieved promising results on the NASA SBIR abstracts.
  • The LLM augmentation consistently improved the F1 score, primarily by enhancing precision, compared to using keywords alone. For instance, at a 1% threshold and k=15, the F1 score was improved by the LLM.
  • The study identified an optimal number of clusters (k=15) for the NASA corpus, balancing label space coverage and redundancy. At k=15, the labels were generally associated with space technologies, while at k=25, the labels became overly generic.
  • The evaluation of redundancy (R) showed distinct regimes based on the number of clusters (k), suggesting that k=8 marked a significant increase in overlap.

Limitations and Future Work

  • Limited Scope: The current study is a pilot on a relatively narrow corpus (NASA SBIR abstracts). The authors suggest extending this to benchmark datasets like Maple and analyzing longer documents (full text) to assess generalizability.
  • Single Label Assignment: The manual annotation task assigned only one label per abstract for rigorous testing. Future work could explore generating multiple labels for a single abstract, which is more reflective of real-world scenarios and was hinted at by the k=25 label set.
  • Weighting Scheme for Multi-Label: Developing a new weighting scheme to handle complex multi-label assignments is a suggested future direction.
  • Broader Applications: The authors envision applications beyond scientific literature, such as creating new industry categories from public company reports or tracking trends in business and public policy.

Summary updated

BibTeX
@inproceedings{sakhrani2024artificial,
  author = {Harsh Sakhrani and Naseela Pervez and Anirudh Ravikumar and Fred Morstatter and Alexandra Graddy-Reed and Andrea Belz},
  title = {{Artificial Intuition: Efficient Classification of Scientific Abstracts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure