Plan before Solving: Problem-Aware Strategy Routing for Mathematical Reasoning with LLMs

Shihao Qi, Jie Ma, Ziang Yin, Lingling Zhang, Jian Zhang, Jun Liu, Feng Tian, Tongliang Liu

TL;DR

This paper introduces PRISM, a novel framework that dynamically selects the best reasoning strategy for Large Language Models (LLMs) tackling mathematical problems. By decoupling strategy planning from execution and learning to predict strategy suitability, PRISM achieves significant accuracy improvements and better efficiency compared to fixed or ensemble approaches.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown promise in mathematical reasoning, but current methods often rely on a single, fixed strategy (like natural language reasoning, code execution, or tool use) or a simple ensemble of strategies. The authors argue that this "one-size-fits-all" approach is suboptimal because different problem types require different reasoning approaches. Furthermore, these methods often overlook the crucial trade-off between accuracy and computational efficiency (speed and resource usage). This leads to either over-reliance on complex, slow methods for simple problems or insufficient rigor for complex ones. PRISM aims to solve this by intelligently routing problems to the most appropriate strategy or combination of strategies based on the problem's characteristics.

Methods

PRISM introduces a two-stage framework:

  1. Offline Training (Strategy Planning):
    • MathStrat Dataset Creation: The authors curate a dataset called "MathStrat" by running four different reasoning strategies (Natural Language Reasoning - NLR, Code-Augmented Reasoning - CAR, Tool-Integrated Reasoning - TIR, and Ensemble-Based Reasoning - EBR) on a large number of math problems. For each problem-strategy pair, they collect metrics on:
  2. Correctness: Did the strategy get the right answer?
  3. Process Quality: Were the reasoning steps valid and efficient?
  4. Computational Efficiency: How fast was it, and how much output did it generate?
  5. Strategy Adapter (SA) Training: A lightweight "Strategy Adapter" model is trained on MathStrat. This adapter learns to predict a "suitability distribution" over the four strategies for any given math problem. It's trained to minimize the difference between its predicted distribution and the target distribution derived from the collected performance metrics.
  6. Online Inference (Targeted Execution):
    • Adaptive Routing Policy: At inference time, the trained Strategy Adapter predicts the suitability of each strategy for the current problem. Based on the confidence of these predictions, PRISM dynamically chooses one of three routing modes:
  7. Confident Routing: If the SA is highly confident in one strategy, it uses only that strategy.
  8. Deliberative Routing: If confidence is high but there's a close competition between the top two strategies, it uses both for verification (e.g., majority voting).
  9. Exploratory Routing: If confidence is low, it uses all available strategies to maximize the chance of finding a correct solution.

This approach decouples the decision-making (planning) from the problem-solving (execution), allowing for dynamic adaptation.

Datasets

The paper uses the following datasets for training and evaluation:

  • MATH: A large-scale, challenging dataset for mathematical reasoning.
  • GSM8K: A dataset of grade-school math word problems.
  • AQUA-RAT: A dataset for evaluating mathematical reasoning abilities.
  • SVAMP: A dataset of math word problems designed to test robustness.
  • ASDiv: A dataset of math problems with diverse difficulty levels.

The authors also created MathStrat, a novel dataset of approximately 13,000 instances specifically for training the Strategy Adapter, containing multi-strategy performance profiles.

Key Insights and Ideas

  • No Single Strategy is Optimal: The experiments clearly show that no single reasoning strategy (NLR, CAR, TIR, EBR) consistently outperforms others across all problem types and datasets. This validates the core motivation for PRISM.
  • Efficiency-Effectiveness Trade-off is Crucial: Simply aiming for the highest accuracy can lead to excessive computational cost. PRISM's adaptive routing allows for balancing these two factors based on problem difficulty and confidence.
  • Predicting Strategy Suitability is Key: The Strategy Adapter, trained on detailed performance metrics, effectively learns to predict which strategy will perform best for a given problem, acting as an intelligent "router."
  • Confidence-Driven Routing is Effective: The three routing modes (Confident, Deliberative, Exploratory) provide a principled way to allocate computational resources. High confidence leads to efficient single-strategy use, while low confidence triggers more robust multi-strategy exploration.
  • Pre-Execution Routing Outperforms Post-Hoc Aggregation: PRISM's approach of selecting strategies before execution is shown to be more effective than methods that generate multiple solutions and then aggregate them (like standard ensemble methods).

Key Results

  • PRISM consistently outperforms individual strategies and ensemble baselines across five mathematical reasoning benchmarks.
  • Improvements range from 0.9% to 7.6% across different base LLMs (Qwen2.5-Math-7B, Deepseek-math-7b-v1, and Llama-3-8B).
  • On the challenging MATH benchmark, PRISM achieved 53.2% accuracy, surpassing the best single strategy (TIR) by 3.1% and the ensemble baseline (EBR) by 2.5%.
  • The adaptive routing mechanism shows significant benefits across diverse model architectures and sizes, demonstrating scalability.
  • The framework is training-free for the LLM itself, meaning it can be applied to any pre-trained model without further fine-tuning.

Limitations and Future Work

  • Threshold Optimization: The performance of the adaptive routing policy is sensitive to the confidence and ambiguity thresholds ($T_c$, $T_a$). While the paper optimizes these on a validation set, finding optimal thresholds for entirely new domains might require further tuning.
  • Generalizability to New Strategies: The current framework is designed for four specific strategies. Extending it to incorporate new or custom reasoning strategies would be a natural next step.
  • Complexity of MathStrat Curation: Creating the MathStrat dataset involves running multiple strategies on many problems, which can be computationally intensive. Future work could explore more efficient ways to curate such preference data.
  • Domain Specificity: While PRISM shows broad applicability, its effectiveness might vary on highly specialized or novel mathematical domains not well-represented in the training data.

The authors suggest that PRISM's ability to dynamically select strategies based on problem characteristics and confidence levels offers a promising direction for improving LLM reasoning capabilities across various domains.

Summary updated

Abstract

Existing methods usually leverage a fixed strategy, such as natural language reasoning, code-augmented reasoning, tool-integrated reasoning, or ensemble-based reasoning, to guide Large Language Models (LLMs) to perform mathematical reasoning. Our analysis reveals that the single strategy cannot adapt to problem-specific requirements and thus overlooks the trade-off between effectiveness and efficiency. To address these issues, we propose Planning and Routing through Instance-Specific Modeling (PRISM), a novel framework that decouples mathematical reasoning into two stages: strategy planning and targeted execution. Specifically, we first curate a multi-strategy preference dataset, which we call MathStrat, capturing correctness, process quality, and computational efficiency for each problem--strategy pair. Then, we train a lightweight Strategy Adapter based on the dataset to obtain confidence distributions over the mentioned four reasoning strategies. At inference time, an adaptive routing policy dynamically tailors the reasoning approach based on predictor confidence. It directs the model to use single-strategy execution for high-confidence predictions, dual-strategy verification for competitive scenarios, or comprehensive multi-strategy exploration for uncertain cases. Extensive experiments across five mathematical reasoning benchmarks demonstrate that PRISM consistently outperforms individual strategies and ensemble baselines, achieving improvements ranging from 0.9% to 7.6% across different base models. The adaptive routing approach shows particularly strong benefits for mathematical reasoning tasks across diverse model architectures. Our code is released at https://github.com/reml-group/PRISM.
BibTeX
@misc{qi2025plan,
  author = {Shihao Qi and Jie Ma and Ziang Yin and Lingling Zhang and Jian Zhang and Jun Liu and Feng Tian and Tongliang Liu},
  title = {{Plan before Solving: Problem-Aware Strategy Routing for Mathematical Reasoning with LLMs}},
  year = {2025},
  eprint = {2509.24377},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24377},
}
Figure