ACL-rlg: A Dataset for Reading List Generation

Julien Aubert-Béduchaud, Florian Boudin, Béatrice Daille, Richard DufourCOLING 2025

TL;DR

This paper introduces ACL-rlg, the largest expert-curated dataset for automatic reading list generation, a crucial task for navigating complex scientific literature. They also define this task as a retrieval problem and present baseline results, revealing that while LLMs like GPT-4o show promise, they also exhibit signs of data contamination and hallucination, highlighting the need for specialized approaches.

What Problem Does This Paper Address?

Navigating the ever-growing volume of scientific literature is a significant challenge for researchers, especially those new to a field. While survey papers offer overviews, they can be broad and lack a structured learning path. Reading lists, curated by experts, provide a more focused and ordered introduction. However, creating these lists is labor-intensive, and existing datasets for automatically generating them are scarce, small, or of questionable quality (often derived from survey papers). This paper addresses this gap by introducing a large, high-quality dataset and a formal framework for evaluating reading list generation.

Methods

The authors frame reading list generation as an article retrieval task. Given a collection of scientific articles and a query from a novice researcher, the goal is to retrieve a concise, ordered list of papers that facilitates understanding a topic. They introduce the ACL-rlg dataset, which consists of 85 expert-curated reading lists derived from tutorials presented at major Natural Language Processing (NLP) conferences. These lists were manually extracted and enriched with metadata. To evaluate their dataset and existing methods, they also developed an annotation protocol for generating queries that novice researchers might use. They then compared various systems, including academic search engines (Semantic Scholar, Google Scholar) and large language models (LLMs) like GPT-4o and Gemini 1.5, using standard information retrieval metrics like Recall@k, NDCG@k, and MRR@k.

Datasets

The primary contribution is the ACL-rlg dataset, featuring 85 expert-curated reading lists from ACL tutorial descriptions, totaling 662 distinct papers. This dataset is significantly larger and of higher quality than previous ones, which were often automatically constructed or very small (e.g., Jardine's 8 lists). They also compare their results against existing datasets like those from Ekstrand et al. (2010), Figueira et al. (2019), and Ding et al. (2022), noting the limitations of those datasets in terms of size, quality, and query standardization.

Key Insights and Ideas

  • Reading list generation is more than just retrieval; order matters. The meaningful ordering of papers is crucial for guiding a reader's learning progression, a factor often overlooked in standard retrieval tasks.
  • LLMs show potential but are not a silver bullet. While GPT-4o demonstrated superior performance compared to traditional search engines, its results raise concerns about data contamination (using tutorial content directly from its training data) and hallucination (generating non-existent references).
  • Query formulation is critical and challenging. The study highlights significant differences in query quality based on annotator expertise, suggesting that simple keyword extraction from titles is insufficient and that understanding user intent is key.
  • Prioritizing highly cited papers can improve results. Employing a "most cited" strategy for search engines significantly boosted performance, indicating that citation count is a strong signal for relevance in reading list generation.

Key Results

  • GPT-4o outperformed other systems, achieving the highest MRR@20 scores, indicating better ranking of relevant articles. However, its overall scores remained relatively low, suggesting the task is still challenging.
  • Traditional search engines (Semantic Scholar, Google Scholar) performed poorly, with low Recall@20 scores, meaning they failed to retrieve a significant portion of the expert-curated articles.
  • LLMs showed signs of data contamination, with a noticeable drop in performance for tutorials from 2023 onwards, a trend not observed in search engines.
  • Approximately 3% of LLM-generated references were "hallucinated" (not found in Semantic Scholar), with this percentage increasing for more recent years.

Limitations and Future Work

  • Dataset Size and Subjectivity: While ACL-rlg is the largest, it's still considered insufficient for training complex supervised models and is best suited for evaluation. The content also reflects the individual interpretations of authors, and the necessity of all listed articles for understanding a field is questioned.
  • Query Discrepancies: Inter-annotator agreement on queries highlighted variations, suggesting a need to understand annotator biases and expertise differences for better query generation systems.
  • Article Substitutability: The paper acknowledges a lack of evaluation on whether predicted articles can truly substitute for expert-recommended ones, which is crucial for accurate generation.
  • Future Directions: The authors suggest exploring Retrieval Augmented Generation (RAG) to mitigate LLM issues like contamination and hallucination. They also emphasize the need for evaluation metrics that specifically account for the ordering of articles within a reading list.

Summary updated

BibTeX
@inproceedings{aubertbeduchaud2025acl,
  author = {Julien Aubert-Béduchaud and Florian Boudin and Béatrice Daille and Richard Dufour},
  title = {{ACL-rlg: A Dataset for Reading List Generation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure