TL;DR
Different math problems demand fundamentally different reasoning strategies—some need natural language proofs, while others require python scripts or external symbolic solvers. Instead of locking a Large Language Model (LLM) into a single reasoning style or wastefully running all of them, PRISM introduces a lightweight "meta-planner" that inspects a problem before execution to dynamically route it to the single best strategy, a dual-strategy verification pass, or a full exploratory ensemble.
What Problem Does This Paper Address?
Current approaches to LLM mathematical reasoning typically commit to a single fixed paradigm:
- Natural Language Reasoning (NLR / Chain-of-Thought): Great for conceptual deduction, but error-prone when handling heavy arithmetic.
- Code-Augmented Reasoning (CAR / Program-Aided Language Models): Deterministic and precise in computation, but brittle during problem formulation.
- Tool-Integrated Reasoning (TIR): Robust for hard multi-step problems, but suffers from high tool orchestration overhead and latency.
- Ensemble-Based Reasoning (EBR / Self-Consistency): Generates multiple expensive trajectories and votes post-hoc.
The authors observe two critical issues:
- No single strategy fits all problems: A strategy like TIR may excel at advanced algebra but underperform simpler code generation on basic word problems.
- Efficiency-effectiveness trade-offs are ignored: Uniformly executing heavy tool loops or large ensembles wastes significant compute on problems that could be solved via a single short chain-of-thought.
Methods
PRISM (Planning and Routing through Instance-Specific Modeling) decouples reasoning into two stages: offline strategy planning and online targeted execution.
- Curating the
MathStratPreference Dataset: The authors ran all four reasoning strategies across ~13,000 math problems. Each trajectory was scored along three axes: - Correctness ($corr \in \{0, 1\}$)
- Process Quality ($qual \in [0, 1]$, penalizing logical fallacies and redundancy)
-
Efficiency ($eff \in [0, 1]$, computed from normalized token length and latency) These metrics were aggregated into a suitability score and converted into a temperature-scaled soft target distribution over strategies.
-
Training the Strategy Adapter (SA): A lightweight 1.5B parameter model is trained to predict strategy suitability distributions using a combination of Kullback-Leibler (KL) divergence (matching the soft profile) and cross-entropy loss (anchoring top-1 prediction).
- Inference-Time Adaptive Routing Policy: Based on the top predicted probability ($p_{\max}$) and runner-up probability ($p_{\text{2nd}}$), the model selects an execution pathway using two tuned thresholds ($\tau_c = 0.4$, $\tau_a = 0.08$):
- Confident Routing ($p_{\max} \ge \tau_c$ and $p_{\max} - p_{\text{2nd}} \ge \tau_a$): High confidence with a clear winner; runs only the single top-ranked strategy.
- Deliberative Routing ($p_{\max} \ge \tau_c$ and $p_{\max} - p_{\text{2nd}} < \tau_a$): High confidence but closely matched contenders; executes the top two strategies and verifies consistency.
- Exploratory Routing ($p_{\max} < \tau_c$): High uncertainty; runs all strategies and aggregates answers via majority voting.
Problem ---> [Strategy Adapter] ---> Distribution {p_NLR, p_CAR, p_TIR, p_EBR}
|
+-------------------------------+------------------------------+
| (High conf, clear lead) | (High conf, close scores) | (Low conf)
v v v
[Confident Mode] [Deliberative Mode] [Exploratory Mode]
Single Strategy Run Top-2 Verification All Strategies Vote
Datasets
- Training: MathStrat, a newly curated dataset of ~13,000 instances built from standard benchmarks (including MATH and GSM8K) annotated with correctness, process quality, and latency profiles across four strategies.
- Evaluation: Evaluated across five benchmark datasets covering elementary math, competition algebra, and word problems:
- MATH500 (Olympiad/high-school level)
- GSM8K (Grade-school arithmetic)
- AQUA-RAT (Algebraic word problems with rationales)
- SVAMP & ASDiv (Diverse math word problems)
- Hungarian Math (used in component ablation studies)
Key Insights and Ideas
- Pre-execution routing beats post-hoc ensembling: Predicting the best strategy beforehand provides a better accuracy–compute trade-off than naively generating multiple answers across paradigms and aggregating them afterwards.
- Weaker models gain more from routing: Relative improvements over the single best strategy were +0.9% on Qwen2.5-Math-7B, +2.9% on Deepseek-Math-7B, and +7.6% on Llama-3-8B. Strategic planning mitigates architectural weaknesses.
- Difficulty dictates discrimination: For simple problems (Levels 1–2), all strategies succeed (confident single-path suffices 90%+ of the time). At moderate difficulty (Levels 3–4), strategy performance diverges sharply (up to a 30% gap between TIR and CAR), making adaptive routing most valuable. At extreme difficulty (Level 5), the system shifts heavily into exploratory multi-path execution.
Key Results
- Superior Benchmark Accuracy: PRISM achieves state-of-the-art results across all evaluated base models, reaching 70.7% average Pass@1 on Qwen2.5-Math-7B (up from 61.9% on TIR and 22.9% on the Hybrid baseline). On MATH500, it achieves 46.2% Pass@1, outperforming TIR (41.4%) and standard Chain-of-Thought (21.2%).
- Balanced Compute Overhead: In efficiency ablations, PRISM reached 33.3% accuracy at 5,600 ms inference time, whereas intermediate heuristic setups consumed 6,000–6,300 ms while achieving only 10.0%–17.8% accuracy.
- Model Scalability: On Qwen2.5 models scaled from 1.5B to 72B parameters, PRISM maintains consistent performance leads on both GSM8K (74.2% $\to$ 95.2%) and MATH500 (32.3% $\to$ 78.4%).
Limitations and Future Work
- Surface-Feature Heuristics: The Strategy Adapter sometimes relies on surface-level keywords (e.g., detecting trigonometry symbols and defaulting to tool usage), overestimating strategy suitability even when the solver fails downstream during symbolic manipulation.
- Tie-Breaking Vulnerabilities: In dual-path mode, ties are broken using the planner's prior confidence rather than runtime validation. If the higher-confidence strategy outputs a floating-point formatting artifact or malformed number while the second strategy is correct, PRISM can choose the wrong answer.
- Future Directions: Incorporating real-time runtime validation signals, tool execution feedback, and richer answer-equivalence normalizers into the decision loop.
Summary updated