Cited Text Spans for Scientific Citation Text Generation

Xiangci Li, Yi-Hui Lee, Jessica OuyangACL 2024

TL;DR

This paper tackles the challenge of generating accurate and faithful scientific citations by moving beyond using only paper abstracts. They propose using "Cited Text Spans" (CTS) – the specific sentences in a cited paper that a citation refers to – as a more grounded input. By developing a scalable, distantly-labeled CTS dataset and a keyword-based retrieval method, they demonstrate that CTS-based citation generation significantly outperforms abstract-based methods, leading to more factual and less hallucinatory citations.

What Problem Does This Paper Address?

Generating accurate and concise descriptions of how one scientific paper relates to another is crucial for academic writing. Current automatic citation generation systems often rely on the abstracts of cited papers. However, this approach has a significant drawback: abstracts may not contain all the relevant information that a citation refers to, leading to "non-factual hallucinations" where the generated citation misrepresents the cited work. This is exacerbated by the fact that full scientific papers are often too long to be processed by standard language models. The core problem is finding a way to ground citation generation in the actual content of the cited paper without requiring expensive manual annotation of the relevant text segments.

Methods

The paper's central innovation is the use of Cited Text Spans (CTS). Instead of relying solely on the abstract, they propose using the specific sentences within a cited paper that directly support the citation.

  1. Distant Labeling for CTS: Manually annotating CTS is prohibitively expensive and time-consuming, and existing datasets suffer from low inter-annotator agreement. To overcome this, the authors develop a "distant labeling" approach. They use the gold citation text itself as a query to rank sentences in the cited paper based on ROUGE recall. This method allows them to create a large-scale, distantly-labeled CTS dataset without manual annotation.
  2. Keyword-Based CTS Retrieval: To address the limitation that using the gold citation text as a query isn't feasible during actual generation, they propose a human-in-the-loop, keyword-based retrieval method. This involves extracting keywords from the target citation text and using these keywords to guide the retrieval of relevant CTS from the cited paper. This approach is practical and aims to mimic how researchers might manually search for relevant information.
  3. Citation Text Generation: They experiment with two generation models: RAG-FiD (Retrieval-Augmented Generation with Fusion-in-Decoder) and LED (Longformer-Encoder-Decoder). These models are conditioned on the retrieved CTS.

Datasets

  • CL-Scisumm and AbuRa'ed: These are existing, manually annotated datasets used for evaluating the quality of their distantly-labeled CTS.
  • CORWA: This dataset, derived from the ACL partition of S2ORC, is used for training and testing their citation text generation models. They create both a large-scale, distantly-labeled training set and use a human-annotated test set.

Key Insights and Ideas

  • Abstracts are Insufficient: The paper strongly argues that relying solely on abstracts for citation generation leads to lower quality and more hallucinations because citations often refer to content beyond the abstract.
  • Distant Labeling is Viable: The research demonstrates that a ROUGE-based distant labeling approach can produce CTS that are of comparable quality to human-annotated ones, making large-scale CTS dataset creation feasible.
  • CTS Retrieval is Key: The core idea is that grounding citation generation in specific "Cited Text Spans" is crucial for accuracy and faithfulness.
  • Human-in-the-Loop is Practical: While fully automatic CTS retrieval is challenging, a keyword-guided approach offers a practical and effective workaround for real-world applications.
  • Human Judges Prefer CTS: Human evaluations show a clear preference for citations generated using CTS over those generated using only abstracts, even when the abstract-based citations are longer.

Key Results

  • Improved Faithfulness and Accuracy: Citation texts generated using CTS (especially Oracle and Keyword) significantly outperform abstract-only generation in terms of token overlap and faithfulness (measured by QuestEval and ANLI).
  • Distant Labeling Performance: The distantly-labeled CTS cover 80-95% of human-labeled CTS and concentrate relevant information within the top 20% of sentences in cited papers.
  • Human Evaluation: Human judges rated CTS-based generated citations higher in relevance and overall quality compared to abstract-only generated citations.
  • Extractive Nature: CTS-based generation tends to be more extractive, meaning it copies more directly from the input, which is seen as a positive for faithfulness and a way to avoid hallucinations, though it raises potential plagiarism concerns.

Limitations and Future Work

  • Challenge of Fully Automatic CTS Retrieval: Retrieving CTS solely from the citation context sentences (without keywords or gold citation text) is challenging due to the semantic mismatch between citing and cited papers. This part of the pipeline requires further investigation.
  • Plagiarism Risk: The extractive nature of CTS-based generation, while good for faithfulness, could lead to plagiarism. The authors suggest post-processing steps like rewriting modules to mitigate this, which are outside the scope of this work.
  • Keyword Prediction: For the human-in-the-loop keyword retrieval, future work could involve automatically predicting these keywords using LLMs or topic analysis to make the process even more user-friendly.
  • Quality Assurance: Large-scale quality assurance of retrieved CTS is difficult due to the cost, and future work could incorporate fact-checking modules.

Summary updated

BibTeX
@inproceedings{li2024cited,
  author = {Xiangci Li and Yi-Hui Lee and Jessica Ouyang},
  title = {{Cited Text Spans for Scientific Citation Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure