Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition

Xuemei Tang, Xufeng Duan, Zhenguang CaiEMNLP 2025

TL;DR

This paper introduces a novel framework to automatically evaluate Large Language Models (LLMs) on their ability to perform key tasks in literature review writing: generating references, writing abstracts, and composing full reviews. Despite advancements, LLMs still struggle with hallucinating references and their performance varies significantly across different academic disciplines.

What Problem Does This Paper Address?

Conducting comprehensive literature reviews is a cornerstone of academic research, but it's a time-consuming and labor-intensive process. LLMs offer a promising avenue to automate parts of this workflow, from identifying relevant papers to summarizing findings. However, there's a significant gap in understanding how well current LLMs actually perform these tasks, especially concerning reliability and accuracy. This paper aims to bridge this gap by providing a systematic and automated evaluation framework for LLM-generated literature review components, moving beyond anecdotal evidence to rigorous assessment.

Methods

The authors propose a three-stage evaluation framework:

  1. Dataset Construction and Task Design: They compiled a dataset of 1,105 literature reviews from 51 journals across six disciplines. Three distinct tasks were designed to assess LLMs:
  2. Reference Generation: Given a title and keywords, LLMs are asked to generate a list of N relevant studies, including specific metadata (title, authors, journal, year, etc.).
  3. Abstract Writing: LLMs are prompted to generate an abstract matching the length of the original, serving as a proxy for literature review planning.
  4. Review Composition: LLMs are tasked with writing a ~1000-word literature review, including citing relevant studies.
  5. LLM-Generated Output Collection: Five LLMs (Claude-3.5-Sonnet, GPT-40, Qwen-2.5-72B, DeepSeek-V3, and Llama-3.2-3B) were used to generate outputs for these tasks.
  6. Assessment of Generated Output: A multidimensional evaluation approach was employed:
  7. Reference Hallucination: Metrics like precision, overlap rate with human-cited references, and title search rate were used to quantify how often LLMs generate fake or inaccurate references.
  8. Factual Consistency: Natural Language Inference (NLI) models were used to assess if the generated text aligns factually with human-written content.
  9. Semantic Coverage: Metrics like cosine similarity, ROUGE, and Key Point Recall (KPR) were used to measure how well the generated content covers the meaning and key points of human-written texts.

Datasets

  • Evaluation Dataset: A newly compiled dataset of 1,105 literature reviews from 51 journals across six disciplines (Biology, Chemistry, Mathematics, Physics, Social Science, Technology) published in 2023 from the Annual Reviews website.
  • External Databases for Reference Verification: Semantic Scholar and Google Scholar were used to verify the existence and accuracy of generated references.

Key Insights and Ideas

  • Hallucinations Persist: Even the most advanced LLMs continue to generate hallucinated references, indicating that reliable reference generation remains a significant challenge.
  • Discipline-Specific Performance: LLM performance in literature review tasks is not uniform; it varies considerably across different academic disciplines. For instance, models tend to perform better in Mathematics but struggle in Chemistry.
  • Review Composition Improves Reference Accuracy: When LLMs are prompted to write a full literature review and cite sources within that context, the accuracy of the generated references improves. This suggests a "mutual constraint" where the writing process itself helps ground the LLM.
  • Multiple Metrics are Crucial: Relying on a single metric is insufficient. Combining metrics like ROUGE, NLI scores, and KPR provides a more comprehensive understanding of LLM capabilities.
  • Human Evaluation Corroborates Automatic Metrics: The study found a strong agreement (kappa of 0.71) between their automatic evaluation method and human assessments, validating the reliability of their framework.

Key Results

  • Reference Generation: Claude-3.5-Sonnet generally outperformed other models in precision, overlap rate, and title search rate. Llama-3.2-3B performed the worst.
  • Abstract Writing: Claude-3.5-Sonnet also led in semantic similarity and factual consistency (TRUE score of 78.10%). DeepSeek-V3 showed strong factual consistency.
  • Review Composition: Similar trends were observed, with Claude-3.5-Sonnet excelling in KPR (recall of claims). Reference accuracy also improved across all models in this task.
  • Disciplinary Performance: In Reference Generation, models showed highest precision in Mathematics and lowest in Chemistry. For Abstract Writing, Social Science consistently yielded the lowest factual consistency.
  • Citation Trends: Generated references were often highly cited, suggesting LLMs may leverage frequently cited works from their training data, leading to more accurate metadata for those specific references.

Limitations and Future Work

  • Limited Scope of Evaluation: The current evaluation primarily compares LLM outputs to human-written texts, focusing on factual consistency and semantic coverage. Future work could incorporate metrics for fluency, topic coverage, and overall review structure coherence.
  • Potential Data Contamination: While efforts were made to mitigate data leakage (e.g., using newer datasets for GPT-5), the possibility of LLM training data overlapping with evaluation data remains a concern. A continuously updated leaderboard is proposed.
  • Abbreviated Names: Handling variations in author names and journal titles required normalization strategies, and minor discrepancies might still exist.
  • Auxiliary Tools for Reference Verification: Reliance on Semantic Scholar for reference verification might lead to incomplete retrieval in some cases.

The authors suggest future work could explore more comprehensive evaluation metrics and continuously monitor LLM performance as models evolve rapidly.

Summary updated

BibTeX
@inproceedings{tang2025large,
  author = {Xuemei Tang and Xufeng Duan and Zhenguang Cai},
  title = {{Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure