Dynamic Strategy Planning for Efficient Question Answering with Large Language Models

Tanmay Parekh, Pradyot Prakash, Alexander Radovic, Akshay Shekher, Denis SavenkovNAACL 2025 · DOI

Abstract

Large Language Models (LLMs) often rely on fixed prompting strategies like Chain-of-Thought (CoT) or Retrieval-Augmented Generation (RAG), which can be inefficient or prone to hallucinations when applied indiscriminately. We present DyPlan (Dynamic Planning), a framework that enables LLMs to dynamically select the most appropriate and cost-effective strategy for a given question. By utilizing a decision-execution-verification pipeline, DyPlan improves accuracy by up to 13% while reducing token usage and retrieval calls by over 30%.

Introduction

In the current LLM landscape, researchers often rely on fixed strategies to improve performance. However, applying a complex strategy to every query is inefficient. Asking an LLM to "think step-by-step" for a simple fact wastes tokens, while relying on internal knowledge for recent events leads to hallucinations.

Figure 4: Examples of different question types requiring different strategies: direct answering for simple facts, reasoning for multi-step problems, and external knowledge for recent events.

The core challenge is creating a model that can assess its own confidence and choose the "cheapest" successful strategy for the task at hand. The authors identify a "hallucination trap": sometimes, forcing a model to reason through a problem it already knows the answer to actually increases the chance of a mistake.

Methodology

The authors introduce two versions of their technique: DyPlan-base and DyPlan-verify. Both rely on a structured pipeline to move away from "one-size-fits-all" prompting.

Figure 6: The DyPlan framework architecture, illustrating the (A) Decision, (B) Execution, and (C) Verification steps for both base and verify versions.

The DyPlan Pipeline

  1. Decision: The model is presented with a "strategy pool" (Direct, Reason, Plan, or Retrieval) and prompted to select the most suitable one based on the question's complexity.
  2. Execution: The model generates the answer using the selected strategy.
  3. Verification (DyPlan-verify): An iterative extension where the model self-evaluates its output. If it lacks confidence, it reverts to the Decision step to select a more robust alternative.

Training via Self-Calibration

To enable this without expensive human labels, the authors used an automated "self-calibration" process. They ran multiple strategies on training data, identified the most efficient successful strategy for each question, and fine-tuned a Llama-3-8B model using these optimal "traces."

Key Insights

  • The "Hierarchy Violation": Traditionally, it is assumed that complex strategies (like RAG) are always superior. However, the authors found that in 7-23% of cases, "simpler" strategies succeeded where complex ones failed due to reasoning errors or irrelevant retrievals.
  • Efficiency as a Metric: This work treats computational cost (tokens and retrievals) as a primary metric alongside accuracy.
  • Self-Knowledge Awareness: Through fine-tuning, the model develops an "internal compass," learning when to rely on its weights versus when to seek external help.

Experimental Results

Evaluated on multi-hop datasets like HotpotQA, 2WikiMultihopQA, and Musique, DyPlan-verify improved F1 scores by an average of 12-13%. Simultaneously, DyPlan-base reduced token usage by 32% and retrieval calls by 26% compared to fixed-strategy baselines. It also significantly outperformed other dynamic frameworks like ReAct and DRAGIN.

Conclusion and Future Work

DyPlan demonstrates that a structured "plan-then-execute" approach is often more efficient than free-form agentic loops. Future work may explore scaling these dynamics to larger models (70B+), expanding the strategy pool to include specialized tools like calculators, and addressing the latency overhead introduced by iterative verification loops.

Summary updated

BibTeX
@inproceedings{parekh2025dynamic,
  author = {Tanmay Parekh and Pradyot Prakash and Alexander Radovic and Akshay Shekher and Denis Savenkov},
  title = {{Dynamic Strategy Planning for Efficient Question Answering with Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.336},
}
Figure