TL;DR
While modern large language models (LLMs) boast context windows exceeding 100K tokens, their ability to reason over long texts degrades sharply compared to simple retrieval. ALR$^2$ addresses this by formulating long-context reasoning as a two-stage "retrieve-then-reason" process, fine-tuning the LLM to extract relevant supporting facts from the context before generating an answer. This explicit alignment slashes retrieval hallucinations from 61.1% down to 0.29% and delivers strong exact-match gains (+8.4 on HotpotQA and +7.9 on SQuAD) up to 128K context lengths.
What Problem Does This Paper Address?
Expanding an LLM's context window does not automatically grant it the ability to reason over that context. In long documents, evidence relevant to a question is often sparse and scattered. When LLMs attempt to jump directly from a 100K-token prompt to an answer, reasoning breaks down—even when the underlying reasoning task itself is trivial.
While intuitively one might solve this by prompting models to "retrieve evidence first, then answer," modern LLMs struggle with this zero-shot: they frequently hallucinate supporting facts or pull incomplete evidence, which cascades into faulty reasoning. The authors investigate how to bridge this gap between long-context retrieval capacity and multi-hop reasoning.
Methods
The authors propose ALR$^2$ (ALigning LLMs with both Retrieval and Reasoning), drawing inspiration from Retrieval-Augmented Generation (RAG):
User Query + Long Context ──► [Stage 1: LLM as Retriever] ──► Extracted Supporting Facts
│
▼
User Query + Extracted Facts ──► [Stage 2: LLM as Reasoner] ──► Final Answer
Unlike classic RAG, ALR$^2$ requires no external vector database or dense retriever; the long context itself acts as the search index on the fly. The LLM serves as both retriever and reasoner:
- Adapted Formulation: The conditional generation probability is broken down into intermediate latent evidence extraction $z$ followed by final generation: $$p_\theta(y \mid q, c) = \sum_z p_\theta(z \mid q, c) \, p_\theta(y \mid z, q, c)$$
- Joint Alignment Objective: Instead of relying purely on prompting, the model is fine-tuned on golden supporting sentences ($z^*$) and final answers ($y^*$): $$\mathcal{L}_{\text{align}} = \frac{1}{N} \sum_{i=1}^N \left( \log p_\theta(z_i^* \mid q_i, c_i) + \log p_\theta(y_i^* \mid z_i^*, q_i, c_i) \right)$$
- Training Setup: Built upon Command-R (35B) with a RoPE base adjusted to $8\times 10^6$. Training contexts span 4K–32K tokens, mixed with pretraining data replay to preserve core capabilities, and evaluated out to 128K via extrapolation.
Datasets
- Preliminary Synthetic Tasks: Passkey retrieval, multi-hop passkey retrieval, and a passkey retrieval + concatenation task embedded in Wikitext-103.
- RULER Benchmark Variants:
- HotpotQA: Multi-hop QA requiring 2–6 supporting facts (5,000 training examples).
- SQuAD: Single-hop QA where the answer sentence serves as the target fact (25,000 training examples).
- Needle-in-a-Haystack (NIAH): Four synthetic variants (Single, Multi-keys, Multi-values, Multi-queries; 1,600 training examples).
- Out-of-Domain Generalization: StrategyQA and TriviaQA (processed following the RULER protocol to test generalization to unseen datasets).
Key Insights and Ideas
- Retrieval is resilient; reasoning is fragile: In synthetic tests, LLMs maintained near-perfect retrieval across 128K contexts, but their accuracy plummeted when asked to perform a simple two-passkey concatenation. The reasoning failure stems from context overload, not an inability to locate information.
- Prompting alone induces severe hallucinations: When unaligned models are prompted to retrieve facts before answering, their retrieval hallucination rate reaches 61.1% on HotpotQA. Unaligned models invent plausible-sounding quotes, derailing subsequent reasoning.
- Explicit alignment tames intermediate representations: By supervising the retrieval step, ALR$^2$ compresses long-context reasoning into a clean, short-context reasoning step over a bulleted list of verified facts, preventing context degradation up to 128K tokens.
Key Results
- HotpotQA: ALR$^2$ scores 76.7 Overall EM, sustaining performance across context lengths (77.2 at 4K to 75.2 at 128K). This outperforms standard fine-tuning without retrieval (CMD-R-FT DA at 68.3 EM) and frontier zero-shot models (GPT-4 at 54.8 EM, Gemini at 56.5 EM).
- SQuAD: ALR$^2$ achieves 72.1 Overall EM, outperforming direct-answer fine-tuning (64.2 EM) and prompted retrieval baselines.
- Retrieval Quality: On HotpotQA, ALR$^2$ slashes the retrieval hallucination rate from 61.1% down to 0.29%, while boosting evidence recall from 34.06% to 68.79%.
- Zero-Shot Transfer: When evaluated on unseen datasets without task-specific fine-tuning, ALR$^2$ outperforms direct fine-tuning by +11.77 EM on StrategyQA (71.51 vs. 59.74) and +2.69 EM on TriviaQA (74.38 vs. 71.69).
Limitations and Future Work
- Task Applicability: The retrieve-then-reason paradigm fits tasks where key evidence is sparse; it is ill-suited for dense information synthesis tasks, such as whole-document summarization, where all context is relevant.
- Fixed Granularity: ALR$^2$ currently retrieves exclusively at sentence-level granularity. The authors note that enabling the model to dynamically adjust retrieval granularity (e.g., phrases or paragraphs) is a valuable direction for future work.
Summary updated