Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QA

Rishabh Maheshwary, Masoud Hashemi, Khyati Mahajan, Shiva Krishna Reddy Malay, Sai Rajeswar, Sathwik Tejaswi Madhusudhan, Spandana Gella, Vikas YadavarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) often struggle with complex, multi-hop questions in Retrieval-Augmented Generation (RAG) settings due to context overload and document noise. We introduce NotesWriting, a plug-and-play approach where a model generates concise, query-focused notes instead of processing raw retrieved text. Evaluated across four benchmarks, including FRAMES and FanoutQA, our method—specifically the ReNAct framework—achieves an average F1 improvement of 15.6 percentage points and reduces input token overhead by tens of thousands of tokens.

Introduction

Retrieval-Augmented Generation (RAG) is essential for grounding LLMs in external knowledge. However, "multi-hop" queries—which require synthesizing information across multiple documents—pose a significant challenge. Standard iterative RAG methods like ReAct or IRCoT often suffer from Context Overload. As the model retrieves more documents at each reasoning step, the context window fills with irrelevant text, leading to distraction, hallucinations, and degraded reasoning performance. Even with long-context models, processing thousands of tokens of raw text for every sub-step is computationally expensive and often degrades reasoning quality.

Methodology: The NotesWriting Framework

To address these issues, we propose NotesWriting, a modular framework that inserts a "note-taker" into the reasoning loop. Instead of passing raw documents to the main reasoning LLM, the system follows a structured cycle:

  1. Query Formulation: The main LLM generates a specific sub-query.
  2. Retrieval: Relevant documents are fetched from the source (e.g., Wikipedia).
  3. Note Extraction: A dedicated LLM extracts only the information relevant to the sub-query.
  4. Aggregation: These notes form a clean "Observation" for the next reasoning step.
Figure 1: The ReNAct workflow, illustrating how the Notes LLM transforms raw Wikipedia pages into concise notes for the reasoning agent.

This approach was integrated into existing frameworks, most notably ReNAct (ReAct + NotesWriting), which explicitly separates reasoning thoughts from the extracted observations. By acting as a high-pass filter, NotesWriting allows the LLM to maintain a clear "mental" model of the problem.

Experimental Evaluation

We evaluated the performance of GPT-4o-mini and Llama 3.1-70B on four challenging benchmarks:

  • FanoutQA: High-entity gathering tasks.
  • FRAMES: Deep multi-hop reasoning (2–15 hops).
  • HotpotQA: Standard multi-hop logic.
  • MultiHop-RAG: News-based generalizability test.

Results and Discussion

The implementation of NotesWriting yielded substantial improvements in both accuracy and efficiency. ReNAct achieved an absolute F1 score increase of 29.1 points on the FRAMES and FanoutQA datasets compared to standard ReAct.

Figure 6: Performance comparison across benchmarks. ReNAct consistently outperforms standard ReAct in F1 scores and GPT-4 evaluation while drastically reducing main LLM input tokens.

Token Efficiency and Noise Reduction

A critical finding was the reduction in "noise." For GPT-4o-mini, input tokens for the main reasoning LLM were reduced by an average of 77K tokens per session. While the note-taking step adds some output token cost, the overall reduction in context window pressure leads to more stable planning and fewer hallucinations. Qualitative analysis showed that the note-taking step helps ground the model, preventing it from getting lost in the irrelevant or redundant information that usually triggers reasoning failures.

Conclusion

NotesWriting demonstrates that "less is more" in complex RAG. By prioritizing concise, query-focused information over raw data, LLMs can navigate deep reasoning chains more effectively. This framework-agnostic module significantly boosts performance while making iterative retrieval more computationally sustainable.

Reference: Maheshwary, R., et al. (2025). Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QA. arXiv:2505.16293v1 [cs.CL].

Summary updated

Abstract

Iterative RAG for multi-hop question answering faces challenges with lengthy contexts and the buildup of irrelevant information. This hinders a model's capacity to process and reason over retrieved content and limits performance. While recent methods focus on compressing retrieved information, they are either restricted to single-round RAG, require finetuning or lack scalability in iterative RAG. To address these challenges, we propose Notes Writing, a method that generates concise and relevant notes from retrieved documents at each step, thereby reducing noise and retaining only essential information. This indirectly increases the effective context length of Large Language Models (LLMs), enabling them to reason and plan more effectively while processing larger volumes of input text. Notes Writing is framework agnostic and can be integrated with different iterative RAG methods. We demonstrate its effectiveness with three iterative RAG methods, across two models and four evaluation datasets. Notes writing yields an average improvement of 15.6 percentage points overall, with minimal increase in output tokens.
BibTeX
@misc{maheshwary2025augmenting,
  author = {Rishabh Maheshwary and Masoud Hashemi and Khyati Mahajan and Shiva Krishna Reddy Malay and Sai Rajeswar and Sathwik Tejaswi Madhusudhan and Spandana Gella and Vikas Yadav},
  title = {{Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QA}},
  year = {2025},
  eprint = {2505.16293},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.16293},
}
Figure