Abstract
Standard Retrieval-Augmented Generation (RAG) systems often struggle with complex scientific queries that contain multiple distinct intents. This paper introduces MuISQA, a novel benchmark for multi-intent scientific question answering, and a corresponding framework designed to enhance retrieval coverage. By leveraging Large Language Models (LLMs) to generate diverse hypothetical answers and decomposing them into atomic factual statements, the MuISQA framework ensures that all facets of a complex query are addressed. Experimental results demonstrate significant improvements in information recall and reasoning accuracy across various scientific domains and general multi-hop datasets.
Introduction
Most current RAG systems operate under a "single-intent assumption," treating a user's question as a search for one specific piece of information. However, scientific inquiry is rarely that simple. A question like "What are ALB gene mutations and their related diseases?" actually contains two distinct intents: identifying specific mutations and linking them to pathologies.
Traditional pipelines tend to retrieve redundant documents for the most dominant part of the query while overlooking others. MuISQA addresses this by explicitly recognizing and retrieving for multiple sub-intents, ensuring a more comprehensive evidence base for the final answer generation.
The MuISQA Framework
The authors propose an intent-aware retrieval framework that moves away from the "one-query-fits-all" approach. The process involves three main stages:
- Hypothetical Query Generation (HQG): Instead of searching with the raw question, the system prompts an LLM to generate several diverse, plausible answers (hypotheses).
- Element Splitting: These long hypothetical answers are decomposed into atomic, intent-specific factual statements. For example, a paragraph about a drug might be split into "Drug X treats Disease Y" and "Drug X has side effect Z."
- RRF-based Reranking: Each atomic statement is used as an independent search query. To merge the results without letting one "loud" query dominate, the authors use Reciprocal Rank Fusion (RRF). RRF prioritizes documents that appear consistently across different intent-specific searches, balancing coverage while reducing redundancy.
The MuISQA Benchmark
To evaluate this approach, the researchers introduced the MuISQA Benchmark, which consists of 500 complex questions across five domains: Biology, Chemistry, Geography, Medicine, and Physics. These are supported by 10,000 documents sourced from open-access arXiv articles. Unlike standard benchmarks that rely on a single "gold" answer span, MuISQA includes multi-intent annotations where each question is paired with multiple sub-intents and their corresponding answers.
To measure success, the authors introduced the Information Recall Rate (IRR), which calculates how many distinct factual units from the "gold" answer were successfully retrieved.
Key Insights and Results
- Hallucination as a Compass: A surprising finding is that "imperfect" hypothetical answers (hallucinations) can actually improve retrieval. Even if the LLM gets a specific fact wrong, the semantic direction of the hypothesis often guides the retriever to the correct neighborhood in the embedding space.
- Performance Gains: The framework consistently outperformed state-of-the-art RAG methods. On the MuISQA Benchmark, it outperformed Naive RAG by roughly 6% in IRR. On the challenging MuSiQue dataset, the method saw a massive +15% improvement in Recall@10 and a +5% boost in F1 score compared to previous SOTA models like ChainRAG.
- Diversity Over Precision: Spreading queries across the embedding space leads to much higher evidence coverage than traditional query rewriting, which often clusters around a single point.
Limitations and Future Work
While effective, the framework introduces computational overhead due to multiple LLM calls and independent retrievals. Furthermore, the quality of retrieval is bottlenecked by the LLM's prior knowledge during the HQG phase. Future research will focus on optimizing the number of generated queries to balance cost and performance, as well as extending the framework to handle multimodal scientific data, such as chemical structures and diagrams.
Summary updated