ESRA: Explainable Scientific Research Assistant

Pollawat Hongwimol, Peeranuth Kehasukcharoen, Pasit Laohawarutchai, Piyawat Lertvittayakumjorn, Aik Beng Ng, Zhangsheng Lai, Timothy Liu, Peerapon VateekulACL 2021

TL;DR

This paper introduces ESRA, an explainable scientific literature discovery platform that goes beyond traditional search by providing explanations for search results and visualizing relationships between concepts. By leveraging a knowledge graph extracted from research paper abstracts, ESRA aims to help researchers understand their queries and discover relevant literature more efficiently.

What Problem Does This Paper Address?

Existing literature search platforms primarily present search results as lists of papers with their metadata. This forces users to manually sift through abstracts to understand the relevance of each paper to their query, a process that is time-consuming and inefficient for expanding their understanding of a research field. While some platforms use knowledge graphs, they often lack the ability to explain why a paper is relevant or to visualize the connections between concepts in a user-friendly way. This gap hinders researchers' ability to quickly grasp the landscape of a topic and explore related ideas.

Methods

ESRA employs a two-pipeline system:

  • Knowledge Graph Construction: This pipeline extracts scientific concepts and their relationships from paper abstracts. It combines two existing models:
  • SpERT: For joint entity and relation extraction.
  • SciIE: For coreference resolution. This combination aims to achieve better performance across all three tasks. The extracted triples are then post-processed to clean duplicates, merge entities, and resolve conflicts before being integrated into a main knowledge graph.

  • Paper Searching Pipeline: This pipeline handles user queries. It first expands the query using similar entities from the knowledge graph. Then, it searches for papers using Elasticsearch, retrieving results based on exact query matches, all keywords, or some keywords. The results are ranked by a combination of Elasticsearch score and citation count. Finally, ESRA generates explanations for each paper using a "Conditional text summarization" technique, presents a "fact list" of related keywords, and offers graph visualizations.

Datasets

  • arXiv dataset (cs.CL category): Used for extracting scientific knowledge to build the knowledge graph. The paper mentions extracting from 23k papers initially, and later states their knowledge graph contains 242k entities and 1.67M relations from the arXiv dataset.
  • SciERC dataset: Used to evaluate the knowledge extraction models due to the lack of ground truth information on the arXiv dataset.

Key Insights and Ideas

  • Explainability is Crucial: The core idea is that simply returning papers isn't enough; users need to understand why a paper is relevant. ESRA's "explanation feature" directly addresses this by providing context-specific summaries.
  • Knowledge Graphs as Bridges: ESRA demonstrates the power of integrating knowledge graphs not just for retrieval, but for enhancing understanding. The graph connects concepts, allowing for richer exploration beyond simple keyword matching.
  • Multi-faceted Exploration: By offering explanations, fact lists, and graph visualizations, ESRA provides users with multiple ways to engage with and understand the literature, catering to different learning styles and needs.
  • Leveraging Pre-trained Models: The system effectively utilizes pre-trained NLP models like T5 for summarization and sentence-BERT for query expansion, showcasing the practical application of state-of-the-art NLP in research tools.

Key Results

  • Knowledge Graph: The constructed knowledge graph contains 242k entities and 1.67M relations from the arXiv cs.CL category.
  • Knowledge Extraction Performance: The combined SpERT+SciIE extractor achieved an F1 score of 70.33 for NER, 50.84 for RE, and 45.87 for CR on the SciERC dataset, performing comparably to SpERT on NER/RE and slightly sacrificing SciIE's performance on CR.
  • Human Evaluation:
    • Explanation: Participants found explanations to be appropriately lengthed (4.44/5) and easy to understand (4.25/5), helping them screen papers faster (4.22/5). Usefulness was rated 3.94/5.
  • Fact List: Rated as helpful for non-NLP users (4.07/5) but less so for NLP specialists (3.78/5), with some comments suggesting redundancy.
  • Graph Visualization: Participants found it helpful for quickly gathering key information (e.g., evaluation metrics), but noted it was difficult to read and suggested layout improvements.
  • Overall average scores for usefulness, understandability, and visual appeal ranged from 3.6 to 4.2, indicating reasonable satisfaction with room for improvement.

Limitations and Future Work

  • Explanation Quality: The paper notes that the T5-generated explanations can sometimes be too similar to the original abstract. Future work could involve incorporating more context into the summarization input.
  • Graph Readability: The graph visualization, while helpful, was found to be difficult to read. Future work should focus on improving node labeling and layout for better readability.
  • Knowledge Graph Refinement: The problem of merged concepts in the knowledge graph (e.g., "recall" and "recall value") is acknowledged as a common weakness that could be alleviated by knowledge graph refinement.
  • Domain Expansion: The current extraction pipeline is optimized for AI-related texts. To extend ESRA to other domains (e.g., life sciences), domain-specific extractors would be needed.
  • Future Work: The authors aim to expand the coverage of their knowledge graph by extracting facts from full documents to enhance ESRA's results.

Summary updated

BibTeX
@inproceedings{hongwimol2021esra,
  author = {Pollawat Hongwimol and Peeranuth Kehasukcharoen and Pasit Laohawarutchai and Piyawat Lertvittayakumjorn and Aik Beng Ng and Zhangsheng Lai and Timothy Liu and Peerapon Vateekul},
  title = {{ESRA: Explainable Scientific Research Assistant}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure