Abstract
Recent advances in large language models (LLMs) demonstrate their impressive reasoning capabilities. However, the reasoning confined to internal parametric space limits LLMs' access to real-time information and understanding of the physical world. To overcome this constraint, we introduce SituatedThinker, a novel framework that enables LLMs to ground their reasoning in real-world contexts through situated thinking, which adaptively combines both internal knowledge and external information with predefined interfaces. By utilizing reinforcement learning, SituatedThinker incentivizes deliberate reasoning with the real world to acquire information and feedback, allowing LLMs to surpass their knowledge boundaries and enhance reasoning. Experimental results demonstrate significant performance improvements on multi-hop question-answering and mathematical reasoning benchmarks. Furthermore, SituatedThinker demonstrates strong performance on unseen tasks, such as KBQA, TableQA, and text-based games, showcasing the generalizable real-world grounded reasoning capability.
Introduction
Despite the impressive reasoning of "long-Chain-of-Thought" (long-CoT) models like OpenAI’s o1 or DeepSeek-R1, they remain largely confined to their internal parametric memory. This "closed-world" nature leads to three major issues: hallucinations, static knowledge that cannot access real-time data, and a lack of physical grounding for tasks involving dynamic environments.
While existing solutions like Retrieval-Augmented Generation (RAG) or tool-calling help, they often rely on rigid, predefined workflows. SITUATEDTHINKER addresses this by introducing situated thinking—the ability for a model to decide when it needs help and how to use external feedback to refine its own internal reasoning.
The SITUATEDTHINKER Framework
The core of the framework is the integration of internal reasoning with external "situated actions." The model uses standardized interfaces to interact with the world, pausing its internal monologue to invoke external tools (like search engines or code executors) when it recognizes a knowledge gap.
To train the model, the authors utilized Group Relative Policy Optimization (GRPO). Notably, the model was not given explicit rewards for using tools. Instead, a simple rule-based reward was applied: Is the final answer correct? Through thousands of trials, the model "discovered" that invoking interfaces to get external data or verify logic was the most reliable path to success.
Experimental Results
SITUATEDTHINKER was trained on a generalist dataset (MuSiQue and Big-Math) and evaluated across diverse benchmarks. It consistently outperformed both base models and standard RAG/tool-calling baselines.
Key performance highlights include:
- Multi-Hop QA: On the Bamboogle benchmark, SITUATEDTHINKER (14B) achieved an Exact Match (EM) score of 0.512, nearly tripling the performance of Naive RAG (0.185).
- Mathematics: On the challenging MATH500 set, it reached 87.1% accuracy, outperforming several math-specialized models despite being a generalist framework.
- Zero-Shot Generalization: The model showed massive performance jumps on tasks it was never trained for, such as the TextWorld environment, where it reached 94% accuracy by effectively using the game's feedback loop.
Key Insights and Discussion
One of the most striking findings is the emergence of knowledge boundaries. The model learns to recognize its own limits, invoking tools specifically when internal parameters are insufficient. Scaling behaviors also emerged: the 14B parameter model used retrieval less often than the 8B model but used code execution more, suggesting that larger models rely more on internal facts but recognize the need for external verification for complex logic.
Limitations and Future Work
Currently, the framework is text-only. Future research will focus on incorporating vision and audio interfaces to enhance real-world grounding. Additionally, extending the framework to handle non-deterministic tasks and open-ended creative reasoning remains a major challenge for moving from text-based tools to actual physical interaction in robotics.
Summary updated