TL;DR
This paper investigates how well Retrieval-Augmented Large Language Models (RALMs) can reason over scientific documents. They found that while RALMs show promise, they tend to "hallucinate" by fabricating evidence to support their predictions, even when trained on scientific corpora. This suggests that simply providing more scientific data doesn't solve the core issue of reliable evidence justification.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful but often generate plausible-sounding but incorrect information, known as hallucinations. Retrieval-Augmented LLMs (RALMs) aim to combat this by retrieving relevant information from external sources to ground their responses. While RALMs have shown success in general knowledge-intensive tasks, their effectiveness and reliability in the specialized domain of scientific document reasoning remain largely unexplored. This paper specifically addresses the question of whether RALMs can accurately extract and justify their reasoning with evidence from scientific documents, and whether training them on scientific data mitigates the risk of fabricating evidence.
Methods
The researchers used the ATLAS model architecture, a state-of-the-art retrieval-based language model. To adapt it for scientific tasks, they employed a multi-stage approach:
- Pretraining: They trained models using either general web data (CC and Wikipedia) or scientific text data (S2ORC).
- Instruction Tuning: They fine-tuned the models on scientific evidence extraction tasks using the SciRepEval benchmark, specifically focusing on "Fields of Study" (FoS) and MAG classification tasks. They designed a specific instruction template to guide the models.
- Evaluation: They evaluated the models on their ability to perform scientific reasoning and, crucially, on the quality of the evidence they used to justify their predictions. They introduced custom metrics for "relevance" and "diversity" of the retrieved evidence.
A key aspect of their method is the use of a "query-side finetuning" approach for the retriever, which keeps the document encoder frozen, making training more efficient.
Datasets
- Pretraining Data:
- Common Crawl (CC) and Wikipedia (for the standard ATLAS model).
- S2ORC (Semantic Scholar Open Research Corpus) scientific text dataset (for the ATLAS-Science model).
- Instruction Tuning & Evaluation Data:
- SciRepEval benchmark: This benchmark provides 25 challenging tasks across various scientific domains. The researchers focused on "Fields of Study" (FoS) and MAG classification tasks.
- S2ORC dataset was preprocessed to create a collection of 354 million text passages, each with a title and up to 512 tokens, covering 19 different scientific domains.
Key Insights and Ideas
- RALMs show promise but aren't a silver bullet: The study confirms that retrieval augmentation generally improves accuracy compared to standard LLMs, even in scientific domains.
- Fabricated evidence is a major concern: The most striking finding is that RALMs, even when performing well on the task itself, often justify their predictions with evidence that is completely irrelevant to the query topic. This suggests a tendency to "hallucinate" evidence.
- Scientific pretraining doesn't solve fabrication: Training models specifically on scientific corpora (like S2ORC) did not significantly alleviate the problem of fabricated evidence. While accuracy saw minor improvements, the relevance and diversity of retrieved evidence remained poor.
- Relevance and diversity are crucial metrics: The paper highlights the importance of evaluating not just the final prediction but also the quality of the supporting evidence. Their custom metrics for relevance and diversity are key to uncovering these issues.
Key Results
- The ATLAS model (retrieval-augmented) outperformed the baseline T5 model in accuracy on both in-distribution (FoS) and out-of-distribution (MAG) tasks.
- ATLAS achieved 84.42% accuracy on FoS and 59.10% on MAG.
- However, the relevance score for retrieved evidence was very low (0.06 for ATLAS), indicating that the passages used to justify predictions were often unrelated to the query topic.
- The diversity score was also very low, suggesting that the model tended to retrieve similar passages across different queries.
- The ATLAS-Science model (pretrained on S2ORC) showed a slight improvement in FoS accuracy (84.70%) but a slight depreciation in MAG accuracy (57.80%) compared to the standard ATLAS. Crucially, its relevance and diversity scores only slightly improved over ATLAS, reinforcing the idea that scientific pretraining alone doesn't fix evidence fabrication.
Limitations and Future Work
- Fabricated Evidence: The primary limitation identified is the tendency of RALMs to fabricate evidence, even when trained on scientific data. This raises concerns about the trustworthiness of these models in scientific applications.
- Future Work: The authors explicitly state their intention to "develop techniques to alleviate these issues in a future work." This suggests a focus on developing methods to ensure that the retrieved evidence is genuinely relevant and supports the model's predictions accurately.
Summary updated