Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models

Yuan Sui, Yufei He, Tri Cao, Sophia Simeng Han, Yulin Chen, Bryan Hooi

TL;DR

Meta-Reasoner is a new framework that empowers Large Language Models (LLMs) to "think about how to think" by dynamically adapting their reasoning strategies during inference. By using Contextual Multi-Armed Bandits (CMABs) to decide when to backtrack, restart, or pivot, the system achieves a 9–12% boost in accuracy while slashing inference time by up to 35% compared to current state-of-the-art methods.

What Problem Does This Paper Address?

While "scaling test-time compute" (giving models more time to "think," as seen in OpenAI’s o1) has led to massive performance gains, it faces two major hurdles: computational inefficiency and error propagation.

Current LLMs often get stuck in "unproductive reasoning trajectories"—loops or dead ends where they continue to generate tokens even after a fundamental mistake has been made. Naively increasing the number of reasoning steps leads to diminishing returns because the model lacks a high-level "advisor" to tell it when a specific approach isn't working. This paper introduces a principled mechanism to holistically revise or redirect the ongoing reasoning process in real-time.

Methods

The authors propose Meta-Reasoner, a framework inspired by dual-process theory in cognitive science. It explicitly decouples low-level stepwise generation (System 1) from high-level strategic selection (System 2).

The system operates in an iterative loop:

  1. Step-wise CoT Generation: The base LLM generates a partial reasoning step.
  2. Progress Reporting: A lightweight summarization function distills the recent reasoning into a concise report. This filters out granular noise, allowing the "meta-reasoner" to focus on high-level progress.
  3. Strategy Selection (The Bandit): The framework treats strategy selection as a Contextual Multi-Armed Bandit (CMAB) problem. Using the LinUCB algorithm, the system evaluates the "context" (the progress report) and selects the best "arm" (strategy) from a pool.
  4. Fixed Strategies: Standard actions like "Backtrack," "Restart," or "Continue."
    • Dynamic Strategies: The system can actually "invent" or refine new strategies on the fly based on the specific problem.
  5. Strategic Guidance: The chosen strategy is fed back to the LLM to guide its next reasoning step.

By using a reward function that balances solution progress (correctness) against resource usage (token cost), the bandit learns to find the most efficient path to a correct answer.

Datasets

The authors evaluated Meta-Reasoner across several reasoning-intensive benchmarks:

  • Mathematics: Game-of-24, TheoremQA (logic/math subsets), Math-500, and the challenging AIME-2024.
  • Science: SciBench (specifically the calculus, statistics, and differential equations subsets).
  • Creative Writing: A task requiring the model to weave four random sentences into a coherent four-paragraph story.

Key Insights and Ideas

  • Decoupling is Efficient: By separating the "doing" from the "planning," the Meta-Reasoner avoids the massive computational overhead of methods like Tree-of-Thoughts (ToT), which expand every possible branch. Instead, it acts as a "strategy controller" that guides a single, adaptive path.
  • Summarization as a Meta-Cognitive Filter: One of the most effective components was the "Progress Report." Distilling long reasoning chains into summaries prevents the meta-reasoner from being distracted by granular details, triggering the LLM's capacity for higher-order reflection.
  • The "Aha" Moment on Efficiency: The authors found that Meta-Reasoner could achieve performance comparable to specialized models like o1-preview but with a 51–55% reduction in inference time. This suggests that how we use compute is just as important as how much compute we use.
  • Domain Agnostic: While tested heavily on math, the system's success in creative writing shows that "meta-reasoning" is a generalizable skill that helps models stay on track regardless of the subject matter.

Key Results

  • Accuracy Gains: Meta-Reasoner outperformed previous SOTA methods (like MACM and Reflexion) by 9–12% across math and science tasks.
  • Speedup: The method reduced inference time by 28–35% under the same compute budget compared to other iterative reasoning frameworks.
  • Model Agnostic: The framework showed consistent improvements across various backbone models, including GPT-4o, Gemini-1206, and open-source models like Qwen3-8B and DeepSeek-R1-Distill.
  • Dynamic Advantage: Using "Dynamic" bandits (where the model can propose new strategies) provided a significant boost over "Fixed" bandits, showing the value of flexible, in-context strategy generation.

Limitations and Future Work

  • Modality Constraints: The current framework is limited to text-based problems. The authors acknowledge that it struggles with tasks requiring other modalities, such as geometry or visual reasoning.
  • Memory Overhead: While it saves on total tokens by avoiding dead ends, the iterative nature of the system adds some memory overhead due to maintaining the reasoning history and multiple API calls.
  • Future Directions: The authors suggest expanding the framework to multi-modal reasoning and exploring more sophisticated reward modeling to further refine the "advisor's" judgment.

Summary updated

Abstract

Large Language Models (LLMs) increasingly rely on prolonged reasoning chains to solve complex tasks. However, this trial-and-error approach often leads to high computational overhead and error propagation, where early mistakes can derail subsequent steps. To address these issues, we introduce Meta-Reasoner, a framework that dynamically optimizes inference-time reasoning by enabling LLMs to "think about how to think." Drawing inspiration from human meta-cognition and dual-process theory, Meta-Reasoner operates as a strategic advisor, decoupling high-level guidance from step-by-step generation. It employs "contextual multi-armed bandits" to iteratively evaluate reasoning progress, and select optimal strategies (e.g., backtrack, clarify ambiguity, restart from scratch, or propose alternative approaches), and reallocates computational resources toward the most promising paths. Our evaluations on mathematical reasoning and puzzles highlight the potential of dynamic reasoning chains to overcome inherent challenges in the LLM reasoning process and also show promise in broader applications, offering a scalable and adaptable solution for reasoning-intensive tasks.
BibTeX
@inproceedings{sui2026meta,
  author = {Yuan Sui and Yufei He and Tri Cao and Sophia Simeng Han and Yulin Chen and Bryan Hooi},
  title = {{Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.649},
}

Loading related content...

Figure