A Study on Leveraging Search and Self-Feedback for Agent Reasoning

K Karthikeyan, Mingzhe Yuan, Elman Mansimov, Katerina Margatina, Anurag Pratik, Daniele Bonadiman, Mahendra K. Sunkara, Yi Zhang, Yassine Benajiba2025 · DOI

Abstract

Karthikeyan K, Michelle Yuan, Elman Mansimov, Katerina Margatina, Anurag Pratik, Daniele Bonadiman, Monica Sunkara, Yi Zhang, Yassine Benajiba. Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025).

2025.

Introduction

Recent breakthroughs in "System 2" reasoning for Large Language Models (LLMs) rely heavily on search algorithms—exploring multiple potential paths to find a solution. Traditionally, these algorithms require a "reward signal" to determine which path is promising. While games like Chess or Go provide clear win/loss rewards, general reasoning tasks rarely offer ground-truth feedback during inference.

The research community has increasingly turned to self-feedback, where the model acts as its own judge. However, the reliability of this internal compass remains under-explored. This study investigates whether LLMs can effectively guide their own search process or if the "blind leading the blind" problem results in performance degradation.

Methodology

The researchers employed the MCTSr (Monte Carlo Tree Search with Self-Refine) framework, applying MCTS to the iterative refinement of answers. They tested several strategies for selecting the final answer from the search tree:

  1. Ground-truth Feedback: An "upper bound" baseline using actual answers to guide search.
  2. Random Selection: Picking a random node from the tree.
  3. Majority Voting: Selecting the answer that appears most frequently across all nodes (consensus).
  4. Maximum Reward: Selecting the answer the model itself gave the highest score.

For complex tool-calling tasks, they introduced Augmented Feedback techniques, including explicit guidelines, in-context learning (ICL), and a specialized hallucination detection module.

Key Results

Mathematical Reasoning (GSM8K)

In mathematical tasks, the study found that the model’s internal "reward score" is often a noisy signal. Majority Voting across the search tree consistently outperformed selecting the answer the model deemed "best." For Llama 3 70B, majority voting raised accuracy from 81.3% (no search) to 88.3%.

Table 1: Performance of various search strategies on the GSM8K benchmark. Majority voting consistently provides the best results without ground-truth feedback.

Tool-Calling (ToolTalk)

In the more complex domain of tool-calling, naive search with self-feedback was found to be actively harmful. On the ToolTalk benchmark, Claude 3 Sonnet’s F1 score plummeted from 0.706 to 0.559 when using MCTS. The search process frequently encouraged the model to hallucinate parameters or make premature tool calls that it then failed to catch during the critique phase.

Table 2: Comparison of 'No Search' vs. 'MCTS' performance on the ToolTalk benchmark. Search significantly degrades performance in complex tool-calling scenarios.

Discussion: The Self-Correction Paradox

While LLMs are often touted for their ability to self-correct, this study highlights a critical "blind spot." Models exhibit a strong bias toward their own generated content and often fail to recognize their own hallucinations as errors during the search process. This suggests that the aggregate "wisdom of the crowd"—even when the crowd consists of multiple samples from a single model—is more reliable than the model’s explicit self-evaluation.

Conclusion

The paper concludes that for autonomous agents to evolve, we must move beyond generic self-feedback. Domain-specific verification mechanisms, such as the hallucination detection modules tested in this study, are necessary to make search viable for complex reasoning tasks. Furthermore, the trade-off between the computational cost of search and the use of larger base models remains a key area for future investigation.

Summary updated

BibTeX
@misc{karthikeyan2025study,
  author = {K Karthikeyan and Mingzhe Yuan and Elman Mansimov and Katerina Margatina and Anurag Pratik and Daniele Bonadiman and Mahendra K. Sunkara and Yi Zhang and Yassine Benajiba},
  title = {{A Study on Leveraging Search and Self-Feedback for Agent Reasoning}},
  year = {2025},
  doi = {10.18653/v1/2025.realm-1.18},
}
Figure