This Reference Does Not Exist: An Exploration of LLM Citation Accuracy and Relevance

Courtni Byun, Piper Vasicek, Kevin SeppiNAACL 2024

TL;DR

Large Language Models (LLMs) like ChatGPT are increasingly used for research tasks, including generating citations. This paper investigates the accuracy and relevance of citations produced by GPT-3, GPT-3.5, and GPT-4 across two different research fields (HCI and NLP). The findings reveal that while GPT-4 generally excels in accuracy, the relevance of its citations is not consistently superior, and performance varies significantly between the two disciplines.

What Problem Does This Paper Address?

Citations are crucial for academic integrity, providing support for claims and guiding readers through existing literature. With the rise of powerful LLMs, researchers are exploring their use in automating citation generation. However, a significant concern is the tendency of these models to "hallucinate" references – generating citations that are inaccurate or entirely fabricated. This paper addresses the need to systematically evaluate how well current LLMs can generate accurate and relevant citations for academic papers, moving beyond just checking if a paper exists to assessing its suitability within a research context.

Methods

The researchers defined three distinct citation recommendation tasks to mimic academic writing scenarios:

  1. Abstract Citations List Task: Models were prompted with a paper's title and abstract and asked to generate a list of 10 relevant citations in APA format.
  2. Abstract Related Works Task: Similar to the first, but the prompt asked the model to write a "Related Works" section, including 10 in-text citations and a bibliography.
  3. Discussion Supported Discussion Task: This task provided the model with the paper's title, abstract, and a portion of its discussion section, asking it to rewrite the discussion with supporting citations.

They compared three LLMs: GPT-3 (text-davinci-003), GPT-3.5-turbo, and GPT-4. To evaluate accuracy, they checked if generated citations referred to real papers and if the authors and publication years were correct. Relevance was assessed by checking for overlap between model-recommended citations/authors and those present in the target papers' bibliographies.

Datasets

The study used a dataset of 20 papers, randomly sampled from two top-tier venues: CHI (Human-Computer Interaction) and EMNLP (Natural Language Processing). For each paper, the title, abstract, and discussion section were extracted. The bibliography of each paper was also collected. This formed the basis for the prompts fed to the LLMs. A total of 1616 annotated citations were generated and analyzed. Google Scholar was used to verify the existence of real papers for the generated citations.

Key Insights and Ideas

  • Discipline Matters: LLMs perform notably better at generating accurate citations for NLP papers compared to HCI papers. This suggests that the training data or the nature of the research in these fields influences LLM performance.
  • Accuracy vs. Relevance: While GPT-4 generally shows higher accuracy in terms of finding real papers and correct authors, its relevance is not consistently superior to older models. This highlights that simply finding a real paper doesn't mean it's a good fit for the context.
  • Hallucinations Persist: Despite improvements, LLMs still struggle with generating perfectly accurate and relevant citations, with issues ranging from incorrect years to fabricated authors.
  • Prompt Engineering is Key: The paper details an extensive prompt evolution process, showing how subtle changes in instructions can impact LLM output, particularly in guiding them towards recent and verifiable sources.

Key Results

  • Title Accuracy: GPT-4 significantly outperformed GPT-3 and GPT-3.5 in title accuracy across both HCI and NLP domains.
  • Discipline Performance: Across most metrics and tasks, models showed better performance on NLP papers than on HCI papers. For instance, title accuracy was consistently higher for NLP.
  • Author Accuracy: GPT-4 generally led in author precision and recall, especially in the "Discussion Supported Discussion" task for HCI papers.
  • Relevance: While GPT-4 showed some advantages in title relevance for NLP papers, the differences in relevance metrics between models were less pronounced, and older models sometimes performed better than GPT-4.
  • Task Performance: The models appeared to struggle more with the "Discussion Supported Discussion" task, possibly due to prompt design challenges.

Limitations and Future Work

The authors acknowledge several limitations:

  • Small Sample Sizes: Due to the breakdown of data by model, task, and discipline, some group sample sizes were below 30, limiting the statistical power of some tests.
  • Limited Citation Features: The study focused on title, author, and year accuracy, not exploring other citation details like page numbers or URLs, where LLM performance might be worse.
  • Manual Curation: Some data collection and curation involved manual checks, introducing a small possibility of human error.
  • Google Scholar Reliance: The reliance on Google Scholar for verification means that papers not appearing on the first page of results might have been incorrectly flagged as fabricated.

Future work could explore more elaborate relevance metrics, investigate the impact of different prompt designs on specific citation features, and examine how LLMs perform with other citation details. The authors also suggest exploring whether changing the prompt from "Discussion" to "Results" might improve performance for NLP papers in the "Discussion Supported Discussion" task.

Summary updated

BibTeX
@inproceedings{byun2024this,
  author = {Courtni Byun and Piper Vasicek and Kevin Seppi},
  title = {{This Reference Does Not Exist: An Exploration of LLM Citation Accuracy and Relevance}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure