When we train a robot to navigate a house or an autonomous vehicle to drive through a city, the traditional approach relies on heavy supervision: feeding the agent thousands of hours of 3D environmental data, geometric maps, and explicit trajectory labels. Yet, humans can walk into a completely unfamiliar building and immediately infer where the kitchen is likely located based on a high-level instruction or a simple glance at the layout.
A rapidly growing body of research demonstrates that Large Language Models (LLMs) and Vision-Language Models (VLMs) already possess this spatial common sense. Even though these foundation models are trained primarily on text and 2D images, the language and visual patterns they absorb reflect the structural topology of the human world. By extracting this latent world knowledge, researchers are building embodied agents capable of complex spatial reasoning and navigation in a completely zero-shot manner—without any environment-specific fine-tuning.
The Latent Topology of Language
Why can a model trained on text tokens reason about physical 3D space? The answer lies in how human language implicitly encodes spatial relationships. When text describes a home, it naturally clusters semantic concepts: refrigerators are found in kitchens, beds are adjacent to nightstands, and room numbers in a hallway typically follow an ascending sequence.
As noted in a comprehensive survey by Yue Zhang et al. (Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models), foundation models have shifted the paradigm of vision-and-language navigation from simple visual-textual pattern matching to reasoning-based planning. Instead of learning rigid mapping policies, agents can leverage internal world knowledge to interpret ambiguous human instructions and navigate complex environments.
For instance, if a navigation agent loses sight of a target landmark, it does not need to explore blindly. Gengze Zhou et al. (NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models) demonstrated that an LLM-powered agent can logically deduce the location of missing landmarks based on semantic context—reasoning that a missing sink is likely located in an unobserved kitchen or bathroom. This internal predictive capability allows LLMs to function as zero-shot text-based world simulators, as explored by Ruoyao Wang et al. (Can Language Models Serve as Text-Based World Simulators?), mapping agent actions directly to expected environmental state changes.
Grounding Text into Physical Constraints
To translate abstract linguistic reasoning into physical movement, the agent must bridge the gap between text tokens and 3D coordinates. A key mechanism for achieving this is the generation of intermediate spatial representations, such as cost maps or visual prompts.
[Natural Language Instruction]
│
▼
┌────────────────────────────────────────┐
│ Large Language Model │
│ (Extracts Semantic Common Sense) │
└────────────┬───────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ Composable 3D Value Map │
│ (Translates Text to Spatial Weights) │
└────────────┬───────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ Low-Level Controller │
│ (Executes Trajectory Planning) │
└────────────────────────────────────────┘
Instead of outputting direct motor commands, the language model acts as a high-level orchestrator that assigns semantic weights to physical space. Wenlong Huang et al. (VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models) showed that LLMs can translate abstract human preferences into specific 3D spatial constraints. By mapping linguistic concepts directly to coordinate grids, the model generates composable value maps that guide low-level controllers in zero-shot manipulation tasks.
Alternatively, when dealing with macro-navigation, the model can interact with the environment through active perception. Sha Zhang et al. (Agent3D-Zero: An Agent for Zero-Shot 3D Understanding) framed 3D perception as an active process where an agent analyzes top-down views of a scene to determine the next optimal camera pose.
This visual grounding is so robust that off-the-shelf VLMs can even interpret human-designed architectural blueprints. David DeFazio et al. (Vision Language Models Can Parse Floor Plan Maps) demonstrated that models like GPT-4o can read floor plans and generate structured robotic navigation plans without any task-specific fine-tuning, directly translating visual architectural data into executable action sequences.
Heuristic Exploration and Scene Imagination
In entirely unexplored environments, an agent cannot rely on a pre-built map. Traditional systems handle this via frontier-based exploration, which treats all unvisited frontiers equally. Zero-shot spatial reasoning changes this by introducing semantic heuristics: the agent prioritizes frontiers that are contextually relevant to its goal.
The framework developed by Wenzhe Cai et al. (InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment) utilizes a multimodal LLM to analyze panoramic visual inputs and provide a heuristic assessment of which directions are most likely to lead to a target object. This leverages the model's internal knowledge of architectural layouts to predict the location of unseen landmarks.
When a physical path is ambiguous, the agent can even "hallucinate" future states to make a decision. Xinxin Zhao et al. (ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination) bypassed explicit 3D semantic mapping entirely by transforming spatial planning into a visual selection task. The VLM is prompted to evaluate a set of synthesized future images against a target object description, selecting the most promising path based on imagination.
┌─► [Synthesized Future Image A] ──► Score: 0.2
│
[Current Observation] ─┼─► [Synthesized Future Image B] ──► Score: 0.8 (Selected)
│
└─► [Synthesized Future Image C] ──► Score: 0.4
This predictive common sense is highly effective for active mapping. In AG-SLAM: Active Gaussian Splatting SLAM, Katrina Ashton et al. used a multimodal LLM as a high-level planner that analyzes top-down renderings of a partial 3D map to identify promising exploration frontiers. Similarly, Zeyuan Yang et al. (RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation) showed that LLMs can predict the semantic layout of unexplored environments by associating detected objects with likely room types, outperforming specialized reinforcement learning policies that require extensive training trajectories.
Closed-Loop Planning and Self-Correction
A core challenge of zero-shot planning is that open-loop execution often fails when encountering physical obstacles or unexpected environmental states. To make zero-shot spatial reasoning viable, modern frameworks implement closed-loop feedback systems where the foundation model dynamically updates its plan based on real-time observations.
Xiwen Liang et al. (CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation) addressed this by introducing a self-correcting mechanism. When the agent encounters an obstacle or realizes an instruction was ambiguous, the LLM re-evaluates the action history and environmental feedback to alter its trajectory online.
To optimize these multi-step decisions without gradient updates, researchers combine the intuitive priors of language models with structured search algorithms. Xiao Yu et al. (Prompt-Based Monte-Carlo Tree Search for Goal-oriented Dialogue Policy Planning) demonstrated that a single large model can simultaneously serve as a policy prior, a value function, and a world model within a Monte Carlo Tree Search (MCTS) framework. This look-ahead simulation allows the agent to evaluate the long-term consequences of its spatial decisions before executing them in the real world.
Overcoming the Long-Tail and the Sim-to-Real Gap
The primary advantage of zero-shot spatial reasoning over supervised learning is its generalization capability. Traditional deep learning models are notoriously fragile when faced with out-of-distribution environments or rare "long-tail" edge cases. Foundation models, by virtue of their massive training data, act as a repository of broad human knowledge that can be applied to novel scenarios.
This capability is particularly valuable in fields like autonomous driving. Can Cui et al. (Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles) and Long Chen et al. (Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving) demonstrated that LLMs can successfully navigate rare or complex driving scenarios by applying general world knowledge—such as anticipating that a ball rolling into the street might be followed by a child—bypassing the limitations of rigid algorithmic controllers.
The emergent nature of this spatial competency is perhaps most vividly illustrated in complex virtual environments. Adrian de Wynter (Will GPT-4 Run DOOM?) showed that an off-the-shelf foundation model, without any specialized training or fine-tuning, can translate raw visual inputs into tactical actions like room navigation, door operation, and combat in a non-deterministic 3D environment.
By treating the physical world not as a collection of geometric coordinates, but as a semantically structured space governed by human common sense, zero-shot spatial reasoning allows embodied agents to cross the sim-to-real gap. They navigate not because they have memorized a specific map, but because they understand the fundamental principles of how human environments are organized.
Key Foundational Work
Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions (2020) — contributed 2 concept elements to this direction.
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022) — contributed 2 concept elements to this direction.