ALR$^2$: A Retrieve-then-Reason Framework for Long-context Question Answering

Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watanabe, Yixuan SuSubmission to ICLR 2025 · Rejected

TL;DR

While modern large language models (LLMs) boast context windows exceeding 100K tokens, their ability to reason over long texts degrades sharply compared to simple retrieval. ALR$^2$ addresses this by formulating long-context reasoning as a two-stage "retrieve-then-reason" process, fine-tuning the LLM to extract relevant supporting facts from the context before generating an answer. This explicit alignment slashes retrieval hallucinations from 61.1% down to 0.29% and delivers strong exact-match gains (+8.4 on HotpotQA and +7.9 on SQuAD) up to 128K context lengths.

What Problem Does This Paper Address?

Expanding an LLM's context window does not automatically grant it the ability to reason over that context. In long documents, evidence relevant to a question is often sparse and scattered. When LLMs attempt to jump directly from a 100K-token prompt to an answer, reasoning breaks down—even when the underlying reasoning task itself is trivial.

While intuitively one might solve this by prompting models to "retrieve evidence first, then answer," modern LLMs struggle with this zero-shot: they frequently hallucinate supporting facts or pull incomplete evidence, which cascades into faulty reasoning. The authors investigate how to bridge this gap between long-context retrieval capacity and multi-hop reasoning.

Methods

The authors propose ALR$^2$ (ALigning LLMs with both Retrieval and Reasoning), drawing inspiration from Retrieval-Augmented Generation (RAG):

User Query + Long Context  ──►  [Stage 1: LLM as Retriever]  ──►  Extracted Supporting Facts
                                                                            │
                                                                            ▼
User Query + Extracted Facts ──► [Stage 2: LLM as Reasoner]   ──►  Final Answer

Unlike classic RAG, ALR$^2$ requires no external vector database or dense retriever; the long context itself acts as the search index on the fly. The LLM serves as both retriever and reasoner:

  1. Adapted Formulation: The conditional generation probability is broken down into intermediate latent evidence extraction $z$ followed by final generation: $$p_\theta(y \mid q, c) = \sum_z p_\theta(z \mid q, c) \, p_\theta(y \mid z, q, c)$$
  2. Joint Alignment Objective: Instead of relying purely on prompting, the model is fine-tuned on golden supporting sentences ($z^*$) and final answers ($y^*$): $$\mathcal{L}_{\text{align}} = \frac{1}{N} \sum_{i=1}^N \left( \log p_\theta(z_i^* \mid q_i, c_i) + \log p_\theta(y_i^* \mid z_i^*, q_i, c_i) \right)$$
  3. Training Setup: Built upon Command-R (35B) with a RoPE base adjusted to $8\times 10^6$. Training contexts span 4K–32K tokens, mixed with pretraining data replay to preserve core capabilities, and evaluated out to 128K via extrapolation.

Datasets

  • Preliminary Synthetic Tasks: Passkey retrieval, multi-hop passkey retrieval, and a passkey retrieval + concatenation task embedded in Wikitext-103.
  • RULER Benchmark Variants:
  • HotpotQA: Multi-hop QA requiring 2–6 supporting facts (5,000 training examples).
  • SQuAD: Single-hop QA where the answer sentence serves as the target fact (25,000 training examples).
  • Needle-in-a-Haystack (NIAH): Four synthetic variants (Single, Multi-keys, Multi-values, Multi-queries; 1,600 training examples).
  • Out-of-Domain Generalization: StrategyQA and TriviaQA (processed following the RULER protocol to test generalization to unseen datasets).

Key Insights and Ideas

  • Retrieval is resilient; reasoning is fragile: In synthetic tests, LLMs maintained near-perfect retrieval across 128K contexts, but their accuracy plummeted when asked to perform a simple two-passkey concatenation. The reasoning failure stems from context overload, not an inability to locate information.
  • Prompting alone induces severe hallucinations: When unaligned models are prompted to retrieve facts before answering, their retrieval hallucination rate reaches 61.1% on HotpotQA. Unaligned models invent plausible-sounding quotes, derailing subsequent reasoning.
  • Explicit alignment tames intermediate representations: By supervising the retrieval step, ALR$^2$ compresses long-context reasoning into a clean, short-context reasoning step over a bulleted list of verified facts, preventing context degradation up to 128K tokens.

Key Results

  • HotpotQA: ALR$^2$ scores 76.7 Overall EM, sustaining performance across context lengths (77.2 at 4K to 75.2 at 128K). This outperforms standard fine-tuning without retrieval (CMD-R-FT DA at 68.3 EM) and frontier zero-shot models (GPT-4 at 54.8 EM, Gemini at 56.5 EM).
  • SQuAD: ALR$^2$ achieves 72.1 Overall EM, outperforming direct-answer fine-tuning (64.2 EM) and prompted retrieval baselines.
  • Retrieval Quality: On HotpotQA, ALR$^2$ slashes the retrieval hallucination rate from 61.1% down to 0.29%, while boosting evidence recall from 34.06% to 68.79%.
  • Zero-Shot Transfer: When evaluated on unseen datasets without task-specific fine-tuning, ALR$^2$ outperforms direct fine-tuning by +11.77 EM on StrategyQA (71.51 vs. 59.74) and +2.69 EM on TriviaQA (74.38 vs. 71.69).

Limitations and Future Work

  • Task Applicability: The retrieve-then-reason paradigm fits tasks where key evidence is sparse; it is ill-suited for dense information synthesis tasks, such as whole-document summarization, where all context is relevant.
  • Fixed Granularity: ALR$^2$ currently retrieves exclusively at sentence-level granularity. The authors note that enabling the model to dynamically adjust retrieval granularity (e.g., phrases or paragraphs) is a valuable direction for future work.

Summary updated

Abstract

The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capability of the model to accurately reason over that context degrades noticeably. This occurs because modern LLMs often become overwhelmed by the vast amount of information in the context; when answering questions, the model must identify and reason over relevant evidence sparsely distributed throughout the text. To alleviate the challenge of long-context reasoning, we develop a retrieve-then-reason framework, enabling LLMs to reason over relevant evidence collected during an intermediate retrieval step. We find that modern LLMs struggle to accurately retrieve relevant facts and instead, often hallucinate "*retrieved facts*", resulting in flawed reasoning and the production of incorrect answers. To address these issues, we introduce ALR$^2$, a method that augments the long-context reasoning capability of LLMs via an explicit two-stage procedure, i.e., aligning LLMs with the objectives of both retrieval and reasoning. We demonstrate the efficacy of ALR$^2$ for mitigating performance degradation in long-context reasoning tasks. Through extensive experiments on long-context QA benchmarks, we find our method to outperform competitive baselines by large margins, achieving at least 8.4 and 7.9 EM gains on the long-context versions of HotpotQA and SQuAD datasets, respectively.
BibTeX
@misc{li2024alr,
  author = {Huayang Li and Pat Verga and Priyanka Sen and Bowen Yang and Vijay Viswanathan and Patrick Lewis and Taro Watanabe and Yixuan Su},
  title = {{ALR$^2$: A Retrieve-then-Reason Framework for Long-context Question Answering}},
  year = {2024},
  eprint = {2410.03227},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03227},
}
Figure