Abstract
Large Language Models (LLMs) often struggle with complex, multi-hop questions in Retrieval-Augmented Generation (RAG) settings due to context overload and document noise. We introduce NotesWriting, a plug-and-play approach where a model generates concise, query-focused notes instead of processing raw retrieved text. Evaluated across four benchmarks, including FRAMES and FanoutQA, our method—specifically the ReNAct framework—achieves an average F1 improvement of 15.6 percentage points and reduces input token overhead by tens of thousands of tokens.
Introduction
Retrieval-Augmented Generation (RAG) is essential for grounding LLMs in external knowledge. However, "multi-hop" queries—which require synthesizing information across multiple documents—pose a significant challenge. Standard iterative RAG methods like ReAct or IRCoT often suffer from Context Overload. As the model retrieves more documents at each reasoning step, the context window fills with irrelevant text, leading to distraction, hallucinations, and degraded reasoning performance. Even with long-context models, processing thousands of tokens of raw text for every sub-step is computationally expensive and often degrades reasoning quality.
Methodology: The NotesWriting Framework
To address these issues, we propose NotesWriting, a modular framework that inserts a "note-taker" into the reasoning loop. Instead of passing raw documents to the main reasoning LLM, the system follows a structured cycle:
- Query Formulation: The main LLM generates a specific sub-query.
- Retrieval: Relevant documents are fetched from the source (e.g., Wikipedia).
- Note Extraction: A dedicated LLM extracts only the information relevant to the sub-query.
- Aggregation: These notes form a clean "Observation" for the next reasoning step.
This approach was integrated into existing frameworks, most notably ReNAct (ReAct + NotesWriting), which explicitly separates reasoning thoughts from the extracted observations. By acting as a high-pass filter, NotesWriting allows the LLM to maintain a clear "mental" model of the problem.
Experimental Evaluation
We evaluated the performance of GPT-4o-mini and Llama 3.1-70B on four challenging benchmarks:
- FanoutQA: High-entity gathering tasks.
- FRAMES: Deep multi-hop reasoning (2–15 hops).
- HotpotQA: Standard multi-hop logic.
- MultiHop-RAG: News-based generalizability test.
Results and Discussion
The implementation of NotesWriting yielded substantial improvements in both accuracy and efficiency. ReNAct achieved an absolute F1 score increase of 29.1 points on the FRAMES and FanoutQA datasets compared to standard ReAct.
Token Efficiency and Noise Reduction
A critical finding was the reduction in "noise." For GPT-4o-mini, input tokens for the main reasoning LLM were reduced by an average of 77K tokens per session. While the note-taking step adds some output token cost, the overall reduction in context window pressure leads to more stable planning and fewer hallucinations. Qualitative analysis showed that the note-taking step helps ground the model, preventing it from getting lost in the irrelevant or redundant information that usually triggers reasoning failures.
Conclusion
NotesWriting demonstrates that "less is more" in complex RAG. By prioritizing concise, query-focused information over raw data, LLMs can navigate deep reasoning chains more effectively. This framework-agnostic module significantly boosts performance while making iterative retrieval more computationally sustainable.
Reference: Maheshwary, R., et al. (2025). Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QA. arXiv:2505.16293v1 [cs.CL].
Summary updated