Abstract
Large Language Models (LLMs) often struggle with tasks requiring precise spatial reasoning and multi-step planning, frequently failing to maintain internal representations of a changing environment. This paper explores a two-stage training pipeline, QWEN-PHYSRL, designed to decouple the acquisition of "spatial physics" from "compositional planning." By first employing Supervised Fine-Tuning (SFT) on atomic spatial transformations and subsequently applying Reinforcement Learning (RL) via Group Relative Policy Optimization (GRPO), we demonstrate that models can achieve superior performance in complex spatial puzzles. Notably, this approach enables "mental tracking" in static environments where the model must internalize state changes without external visual updates.
1. Introduction
While LLMs have demonstrated remarkable proficiency in linguistic tasks, their ability to perform spatial reasoning—such as mental rotation, translation, and navigation—remains a significant bottleneck. Standard models often hallucinate object positions or fail to predict the outcome of sequential transformations. This deficiency stems from the dual nature of spatial reasoning: it requires both an understanding of the underlying "physics" (the rules of transformation) and the ability to plan a sequence of actions to reach a goal.
Most current approaches either rely on external symbolic solvers or attempt to teach the model everything at once through Reinforcement Learning. The former limits the LLM's autonomy, while the latter is often unstable and fails to build a deep internal representation of space. This work proposes a modular approach to bridge this gap.
2. Methodology: The QWEN-PHYSRL Pipeline
Researchers from Purdue University propose a two-stage training framework to systematically build spatial competence.
2.1 Stage 1: Supervised Fine-Tuning for Spatial Physics
The initial phase focuses on teaching the model the "atomic" building blocks of spatial movement. Using a base Qwen2.5-1.5B model, the authors performed SFT on a dataset of 12,000 single-step transformations. These transformations included rotation, translation (up, down, left, right), and scaling. This stage ensures the model understands the fundamental consequences of individual actions—the "physics" of its environment—before attempting complex sequences.
2.2 Stage 2: Reinforcement Learning for Planning
In the second stage, the physics-aware model is frozen, and lightweight LoRA adapters are introduced. These adapters are trained using Group Relative Policy Optimization (GRPO), the same RL framework utilized by DeepSeek-R1. During this phase, the model learns to compose the previously learned atomic actions into a strategic sequence to transform a starting ASCII shape into a target configuration. The training occurs in a closed-loop environment, providing iterative feedback on the model's progress toward the goal.
3. Experimental Design
The study utilizes a synthetic ASCII-art environment where shapes (represented by #) and boundaries (*) are processed as sequences of tokens. This setup forces the model to derive spatial relationships purely from text-based representations.
Evaluation scenarios consisted of 100 unseen random puzzles requiring up to 5 steps to solve. A critical component of the evaluation is the distinction between two settings:
- Dynamic Setting: The environment updates the ASCII map after every move, providing the model with a fresh visual state.
- Static Setting: The map remains unchanged after the initial prompt. The model must perform "mental tracking," maintaining an internal state of the world as it executes a sequence of moves.
4. Results and Analysis
The QWEN-PHYSRL model demonstrated significant improvements over end-to-end RL baselines across all metrics.
4.1 Performance and Mental Tracking
In the Dynamic setting, QWEN-PHYSRL achieved a reward of 2.457, approaching the theoretical maximum of 2.7. In contrast, the end-to-end RL model achieved only 1.626. The most striking results appeared in the Static setting; while the end-to-end model's performance collapsed to -0.216 (performing worse than a random policy), QWEN-PHYSRL maintained a high reward of 1.717. This suggests that the SFT stage successfully instilled a "mental model" of space that the model could update internally.
4.2 Attention Mechanism Shifts
Analysis of the model's internal attention heads revealed that the two-stage training fundamentally altered how the model processes spatial data. The physics-aware model showed increased attention toward spatially informative tokens (the # characters) in its middle layers, indicating a more sophisticated internal representation of the shape's structure compared to the base model.
5. Conclusion and Future Work
The success of QWEN-PHYSRL highlights that decomposing spatial reasoning into "rules" and "planning" is essential for training stable and capable LLMs. By mastering the alphabet of spatial physics before attempting the sentences of complex planning, the model achieves a level of mental tracking previously unattainable by standard RL methods.
Future research will look to expand the action space beyond discrete 90-degree rotations and grid movements into continuous 3D environments. Additionally, applying these principles to Multi-modal Large Language Models (VLMs) could bridge the gap between text-based spatial reasoning and real-world visual navigation.
Summary updated