NLP-KG: A System for Exploratory Search of Scientific Literature in Natural Language Processing

Tim Schopf, Florian MatthesACL 2024

TL;DR

Navigating the ever-growing sea of Natural Language Processing (NLP) research can be daunting, especially for those new to a specific sub-field. This paper introduces NLP-KG, a system designed to make exploratory literature search in NLP more intuitive and effective. It combines a knowledge graph of NLP concepts with advanced search and conversational AI features to help researchers discover relevant papers, understand unfamiliar topics, and explore connections between different areas of NLP.

What Problem Does This Paper Address?

The explosion of NLP research means that finding relevant papers, especially when exploring new or unfamiliar sub-fields, is increasingly challenging. Traditional keyword-based search engines often return overwhelming results, many of which are irrelevant, and they don't effectively support the "exploratory" nature of learning about a new area. Researchers need tools that go beyond simple keyword matching to help them understand the landscape of a field, identify key concepts, and discover foundational or survey papers. Existing systems, while useful for general academic search, lack the specialized focus and advanced features needed for deep dives into NLP literature.

Methods

NLP-KG employs a multi-faceted approach to tackle this problem:

  • Knowledge Graph (KG) Construction: The core of NLP-KG is a hierarchical knowledge graph of Fields of Study (FoS) in NLP. This graph is built semi-automatically by extracting entities and relations from a large corpus of NLP publications (ACL Anthology and arXiv cs.CL). They fine-tuned models like SciBERT for entity and relation extraction and used GPT-4 for generating textual descriptions of FoS.
  • FoS Classification: To link publications to the KG, they developed a two-step classification approach. The first step uses a pre-trained model for high-level FoS classification, and the second step uses a rule-based approach for finer-grained classification within the hierarchy.
  • Survey Paper Classification: A dedicated binary classifier was trained to identify survey papers, enabling users to quickly get an overview of a topic.
  • Semantic Search: NLP-KG uses a hybrid approach combining sparse (BM25) and dense (SPECTER2 embeddings) retrieval methods, merged using Reciprocal Rank Fusion (RRF). This is further enhanced by a reranking module that uses NLP-KG metadata.
  • Conversational Search & "Ask This Paper": Leveraging Large Language Models (LLMs) like GPT-4, NLP-KG offers conversational search. Users can ask questions in natural language, and the system retrieves relevant papers using semantic search, then uses a Retrieval Augmented Generation (RAG) pipeline to provide grounded answers with inline citations. The "Ask This Paper" feature allows users to ask specific questions about the content of individual publications.

Datasets

  • Corpus for KG Construction: Titles and abstracts from the ACL Anthology and the arXiv cs.CL category (totaling 116,053 documents).
  • FoS Extraction Training: A slightly adapted SciERC dataset was used for fine-tuning entity and relation extraction models.
  • FoS Classification Training: A pre-trained model from Schopf et al. (2023) was used for initial classification.
  • Survey Paper Classification Dataset: A custom dataset was created, comprising 787 annotated survey papers and 11,805 non-survey papers in NLP, derived from keyword searches and manual annotation.
  • Conversational Search Evaluation: 50 randomly generated NLP-related questions were used.

Key Insights and Ideas

  • The Power of Hierarchical Knowledge: Building a structured hierarchy of NLP Fields of Study is crucial for enabling intuitive exploration. Users can navigate from broad concepts to specific ones, mirroring how one might learn about a new field.
  • Hybrid Retrieval for Robustness: Combining keyword-based and embedding-based search (BM25 + SPECTER2 + RRF) provides a more comprehensive and accurate retrieval system than either method alone.
  • LLMs as Exploratory Companions: Integrating LLMs for conversational search and answering specific paper questions significantly enhances the user experience, making complex literature more accessible and understandable. The RAG approach is key to grounding LLM responses in factual evidence from the papers.
  • Survey Papers as Entry Points: Automatically identifying and filtering for survey papers is a highly effective strategy for introducing users to new research areas.

Key Results

  • FoS Hierarchy Evaluation: The constructed FoS hierarchy graph achieved high precision (99.95%), recall (99.65%), and F₁ (99.80%) scores in a user study, with a low Mean Absolute Percentage Error (MAPE) of 0.478, indicating efficient navigation.
  • RAG Performance: The conversational search pipeline, using both GPT-3.5 and GPT-4, demonstrated strong performance with high faithfulness (0.9661-0.9714) and answer relevance (0.8479-0.8670) scores, indicating that answers are grounded in the provided context and address the user's questions.
  • Feature Comparison: NLP-KG stands out by offering NLP-specific features, a FoS hierarchy, survey filters, and LLM-powered conversational and "Ask This Paper" functionalities, which are largely absent in general academic search engines like Google Scholar or Semantic Scholar.

Limitations and Future Work

  • Domain Expert Bias: The construction of the FoS hierarchy graph is influenced by the choices of domain experts, which could introduce bias. While they attempted to mitigate this by using automated extraction and aligning opinions, inherent subjectivity remains.
  • Limited Data Coverage: The system's database is currently limited to papers from the ACL Anthology and arXiv cs.CL. NLP research is also published in other major conferences (e.g., AAAI, NeurIPS, ICLR, ICML), which are not yet included.
  • LLM Hallucinations: Despite efforts to ground LLM responses in retrieved knowledge, LLMs can still make mistakes or "hallucinate." Users are advised to always verify important information.
  • Computational Cost: LLMs are computationally expensive, requiring significant resources.

Future work could involve expanding the database to include a wider range of NLP publications, further refining the FoS hierarchy, and exploring more advanced LLM integration techniques.

Summary updated

BibTeX
@inproceedings{schopf2024nlp,
  author = {Tim Schopf and Florian Matthes},
  title = {{NLP-KG: A System for Exploratory Search of Scientific Literature in Natural Language Processing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure