SciLit: A Platform for Joint Scientific Literature Discovery, Summarization and Citation Generation

Nianlong Gu, Richard H.r. HahnloserACL 2023

TL;DR

Writing research papers is time-consuming, especially when it comes to finding relevant literature, summarizing it, and citing it correctly. SciLit is a new platform that automates these tasks by recommending papers, extracting key highlights, and even generating citation sentences, all tailored to your specific context and keywords. This could significantly speed up the research writing process for scientists.

What Problem Does This Paper Address?

The process of conducting a literature review, summarizing findings, and integrating them into a manuscript is a cornerstone of scientific research. However, it's a labor-intensive task. Researchers often spend considerable time sifting through vast databases of scientific papers, identifying relevant ones, extracting their core contributions, and then crafting citations that seamlessly fit into their own writing. This manual process is not only time-consuming but also prone to errors or omissions. The authors of SciLit aim to alleviate this burden by developing an end-to-end system that leverages natural language processing (NLP) to streamline these critical steps.

Methods

SciLit is built around a modular, microservice-based architecture, allowing for easy updates and scalability. Its core workflow involves three main stages:

  • Literature Discovery: This module recommends relevant papers based on user-provided context (text surrounding a citation) and keywords. It employs a two-stage approach:
  • Prefetching: A hierarchical system uses a combination of keyword-based inverted indexes and embedding-based nearest neighbor search to quickly retrieve a large set of candidate papers from massive corpora.
  • Reranking: A fine-tuned SciBERT model then reranks these candidates to identify the most relevant ones, considering the user's query.
  • Extractive Summarization: For each recommended paper, SciLit extracts a short list of key sentences (highlights) from its full text. This is done using an extractive summarization model (MemSum) to avoid potential hallucinations common in abstractive methods.
  • Citation Generation: This module takes the user's context, keywords, and the abstract of a target paper to generate a concise citation sentence. This sentence is designed to seamlessly integrate into the user's manuscript, referencing the target paper and aligning with the provided context and keywords. A fine-tuned T5 model is used for this task.

The system is designed to be flexible, allowing users to input keywords to guide the search and citation generation, making the process more interactive and tunable.

Datasets

The system is trained and evaluated on large scientific corpora, including:

  • S2ORC: A massive corpus of scientific papers.
  • PMCOA (PubMed Central Open Access): A collection of biomedical literature.
  • arXiv: A popular repository for preprints in physics, mathematics, computer science, and other fields.

The authors also mention using the PubMed dataset for training their summarization model and an evaluation dataset of 1530 samples primarily from 2022 papers in computer science and biomedical science for assessing SciLit's performance.

Key Insights and Ideas

  • The Power of Keywords: The authors emphasize that keywords are crucial for effective citation generation. They argue that keywords often convey an author's specific intention more directly than a long text, allowing for more targeted and relevant results.
  • Integrated System Advantage: SciLit's strength lies in its integrated approach. By combining literature retrieval, summarization, and citation generation, it offers a holistic solution that outperforms separate "generation-only" pipelines, suggesting that good paper retrieval significantly boosts citation quality.
  • Extractive Summarization for Reliability: The choice of extractive summarization over abstractive methods for highlights is a deliberate decision to ensure factual accuracy and avoid generating misleading information.
  • Microservice Architecture for Agility: Building SciLit as a network of microservices allows for dynamic updates and easy integration of new algorithms or data sources, making the platform adaptable to the rapidly evolving NLP landscape.

Key Results

  • Paper Retrieval: The system achieves good recall rates for recommended papers, with optimal performance observed when a moderate number of pre-fetched candidates (Np=100) are used.
  • Extractive Summarization: MemSum, the chosen summarization model, significantly outperforms BertSum, achieving high ROUGE F1 scores and summarizing papers efficiently (0.1 sec per paper).
  • Citation Generation: The "Best-of-top-K" pipeline (retrieval followed by generation) significantly outperforms a "generation-only" pipeline, demonstrating the value of the retrieval component. For K=5 and 10, substantial improvements in ROUGE scores were observed.
  • Ablation Studies: Removing keywords leads to a significant degradation in both literature retrieval and citation generation performance, underscoring the utility of keywords.

Limitations and Future Work

  • Limitations: The paper acknowledges that the performance of the reranking component can be impacted by an excessive number of pre-fetched candidates. The current system focuses on generating citations for single-paper references.
  • Future Work: The authors plan to further improve the performance of each module, particularly the citation generation component. They also aim to collect user feedback to enhance the overall workflow and the frontend user experience.

Summary updated

BibTeX
@inproceedings{gu2023scilit,
  author = {Nianlong Gu and Richard H.R. Hahnloser},
  title = {{SciLit: A Platform for Joint Scientific Literature Discovery, Summarization and Citation Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure