Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications

Ethan Lin, Zhiyuan Peng, Yi FangNAACL 2025

TL;DR

Large Language Models (LLMs) are increasingly used to analyze scientific literature, but their ability to assess "novelty" – a crucial factor in scientific discovery – remains underexplored. This paper introduces SchNovel, a new benchmark for evaluating LLMs on this task, and proposes RAG-Novelty, a retrieval-augmented method that significantly improves LLM performance by grounding novelty assessments in relevant prior work.

What Problem Does This Paper Address?

Identifying novel contributions is a bottleneck in scientific discovery. It helps prioritize research, avoid redundant efforts, and guide future investigations. While LLMs excel at many language tasks, their capacity to systematically assess novelty in scholarly publications, where contributions build incrementally on existing work, is not well understood. Existing benchmarks for creativity often focus on semantic novelty (e.g., generating multiple uses for an object) and are not directly applicable to the nuanced, context-dependent nature of scientific novelty. This paper aims to bridge this gap by creating a dedicated benchmark and a method to improve LLM performance in this critical area.

Methods

The authors introduce SchNovel, a benchmark designed to evaluate LLMs' ability to assess novelty in scholarly publications. It consists of 15,000 pairs of papers from six fields (Computer Science, Mathematics, Physics, Quantitative Biology, Quantitative Finance, and Statistics) sampled from the arXiv dataset. For each pair, the more recently published paper is assumed to be more novel.

To enhance LLM performance, they propose RAG-Novelty, a retrieval-augmented generation method. This approach mimics how human reviewers might assess novelty: by first retrieving similar papers to provide context. Specifically, the abstract of a query paper is used to retrieve top-K similar papers from an indexed corpus of prior work. This retrieved context, along with the original papers' titles and abstracts, is then fed to the LLM to make a novelty assessment. The authors explored various prompting strategies, finding that a self-reflection prompt, augmented with retrieved paper information and contextual data analysis (averaging publication dates of retrieved papers), yielded the best results.

Datasets

The primary dataset used is the arXiv dataset, from which approximately 2.5 million articles are available. For the SchNovel benchmark, the authors sampled 15,000 paper pairs across six fields. The papers were selected with publication date gaps ranging from 2 to 10 years.

Key Insights and Ideas

  • Scientific novelty is context-dependent: Unlike general creativity, scientific novelty is deeply tied to the existing body of knowledge and its evolution over time. Simply generating a "new" idea isn't enough; it must be novel within the current research landscape.
  • LLMs struggle with scientific novelty out-of-the-box: Standard LLMs, even powerful ones, often exhibit biases (like position bias) and lack the contextual understanding to accurately assess scientific novelty based solely on titles and abstracts.
  • Retrieval augmentation is key: Mimicking human peer review by providing LLMs with relevant prior work significantly boosts their ability to assess novelty. This grounding helps them understand how a paper fits into the broader scientific narrative.
  • Metadata can introduce bias: Surprisingly, adding author affiliation to the prompt introduced bias, with LLMs showing a tendency to favor papers from perceived "top research universities." This highlights the need for careful consideration of how LLMs process information beyond core content.

Key Results

  • RAG-Novelty outperforms baselines: The proposed RAG-Novelty method achieved the highest accuracy in assessing scholarly novelty, significantly outperforming baseline methods like Zero-Shot, Two-Shot, Self-Reflection, LLM Discussion, and Self-Consistency.
  • Pairwise comparison is superior: Evaluating papers in pairs (pairwise comparison) yielded much better results than evaluating them individually (pointwise comparison), highlighting the importance of direct contextual comparison.
  • Field-specific performance varies: The Computer Science field showed the highest accuracy, likely due to more standardized metrics and data availability. Mathematics and Physics showed lower accuracy, potentially due to complex notation and domain-specific challenges.
  • Year gap matters more than start year: The model's ability to evaluate novelty was more dependent on the time gap between papers than the specific publication years.
  • LLM performance varies: GPT-4o-mini consistently outperformed other LLMs tested, including GPT-3.5, LLaMA 3.1-8b, Mistral 7b, and Gemma 2-9b. Smaller models exhibited more pronounced position bias.
  • Adding TLDRs decreased accuracy: Surprisingly, incorporating TLDR summaries generated by SciTLDR slightly decreased accuracy, though it did help mitigate position bias.

Limitations and Future Work

  • Abstract-only evaluation: The study relied on titles and abstracts, which may not fully capture the novelty of research compared to the complete text. Abstracts can omit crucial technical details.
  • arXiv dataset limitation: The use of only arXiv data limits the generalizability. Future work should include peer-reviewed publications and papers from more diverse sources.
  • Future Directions:
    • Expand SchNovel with more papers and fields.
  • Investigate which specific parts of a paper are most indicative of novelty.
  • Further study LLM biases, particularly affiliation bias, and explore methods to mitigate them.
  • Explore how LLMs process affiliation information.

Summary updated

BibTeX
@inproceedings{lin2025evaluating,
  author = {Ethan Lin and Zhiyuan Peng and Yi Fang},
  title = {{Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure