Abstract
Karthikeyan K, Michelle Yuan, Elman Mansimov, Katerina Margatina, Anurag Pratik, Daniele Bonadiman, Monica Sunkara, Yi Zhang, Yassine Benajiba. Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025).
2025.
Introduction
Recent breakthroughs in "System 2" reasoning for Large Language Models (LLMs) rely heavily on search algorithms—exploring multiple potential paths to find a solution. Traditionally, these algorithms require a "reward signal" to determine which path is promising. While games like Chess or Go provide clear win/loss rewards, general reasoning tasks rarely offer ground-truth feedback during inference.
The research community has increasingly turned to self-feedback, where the model acts as its own judge. However, the reliability of this internal compass remains under-explored. This study investigates whether LLMs can effectively guide their own search process or if the "blind leading the blind" problem results in performance degradation.
Methodology
The researchers employed the MCTSr (Monte Carlo Tree Search with Self-Refine) framework, applying MCTS to the iterative refinement of answers. They tested several strategies for selecting the final answer from the search tree:
- Ground-truth Feedback: An "upper bound" baseline using actual answers to guide search.
- Random Selection: Picking a random node from the tree.
- Majority Voting: Selecting the answer that appears most frequently across all nodes (consensus).
- Maximum Reward: Selecting the answer the model itself gave the highest score.
For complex tool-calling tasks, they introduced Augmented Feedback techniques, including explicit guidelines, in-context learning (ICL), and a specialized hallucination detection module.
Key Results
Mathematical Reasoning (GSM8K)
In mathematical tasks, the study found that the model’s internal "reward score" is often a noisy signal. Majority Voting across the search tree consistently outperformed selecting the answer the model deemed "best." For Llama 3 70B, majority voting raised accuracy from 81.3% (no search) to 88.3%.
Tool-Calling (ToolTalk)
In the more complex domain of tool-calling, naive search with self-feedback was found to be actively harmful. On the ToolTalk benchmark, Claude 3 Sonnet’s F1 score plummeted from 0.706 to 0.559 when using MCTS. The search process frequently encouraged the model to hallucinate parameters or make premature tool calls that it then failed to catch during the critique phase.
Discussion: The Self-Correction Paradox
While LLMs are often touted for their ability to self-correct, this study highlights a critical "blind spot." Models exhibit a strong bias toward their own generated content and often fail to recognize their own hallucinations as errors during the search process. This suggests that the aggregate "wisdom of the crowd"—even when the crowd consists of multiple samples from a single model—is more reliable than the model’s explicit self-evaluation.
Conclusion
The paper concludes that for autonomous agents to evolve, we must move beyond generic self-feedback. Domain-specific verification mechanisms, such as the hallucination detection modules tested in this study, are necessary to make search viable for complex reasoning tasks. Furthermore, the trade-off between the computational cost of search and the use of larger base models remains a key area for future investigation.
Summary updated