In a 3D point cloud, there is no "left" or "right." There are only X, Y, and Z coordinates. Yet, when humans interact with an environment, we rarely use absolute coordinates. We say, "The keys are on the table to your left," or "Walk past the chair and turn right." For an AI agent to understand these instructions, it must perform a "spatial gymnastics" known as situated reasoning: the ability to transform a global, objective 3D map into a local, subjective perspective.
The Perspective Gap
Most traditional 3D vision models treat environments as allocentric—a "god's eye view" where objects exist in a fixed global coordinate system. While this is sufficient for object detection, it fails in interactive scenarios. As Xiaojian Ma and colleagues demonstrate in SQA3D: Situated Question Answering in 3D Scenes, there is a massive performance gap between humans and AI when questions are conditioned on a specific "situation."
A "situation" consists of a position and a heading (orientation). Without knowing which way the agent is facing, a command like "look at the painting behind you" is mathematically undefined. SQA3D forces models to first localize themselves based on a textual description of their surroundings before they can even attempt to answer a spatial query.
Transforming the World Frame
The core mechanism of situated reasoning is the transformation of the world coordinate system to be agent-centric. Instead of the origin $(0,0,0)$ being an arbitrary corner of a room, the origin becomes the agent's current location, and the axes align with the agent's forward-facing direction.
In Situational Awareness Matters in 3D Vision Language Reasoning, Liangyan Gui and Yu-Xiong Wang introduce SIG3D, which explicitly estimates this transformation. By re-encoding the 3D environment from the perspective described in a text prompt, the model bridges the gap between raw point clouds and linguistic spatial relations. Similarly, Jiangyong Huang shows in MSQA that normalizing 3D coordinates to the agent's viewpoint significantly simplifies the geometric calculations required for the model to understand relative directions.
Resolving Ambiguity with Anchors
Even with a defined perspective, 3D space is "fuzzy." Sparse point clouds lack the clear boundaries of 2D images. To resolve this, researchers use "anchors"—stable reference points that ground the spatial reasoning process.
- Object Anchors: Li Yang and the authors of CORE-3DVG suggest that humans describe scenes by linking targets to nearby landmarks (e.g., "the lamp next to the sofa"). By explicitly detecting these "anchor" objects, models can calculate relative offsets more accurately.
- Multi-Key Anchors: The MiKASA framework expands this by using multiple local anchors. This mimics how we navigate: we don't just use one reference point; we triangulate our position based on several landmarks simultaneously.
- Visual Anchors: Sometimes, the best way to resolve 3D ambiguity is to look at a 2D projection. Qi Chen in IPDN argues that 2D images provide a stable reference frame that "anchors" the 3D point cloud, making it easier to distinguish "left" from "right" in a 360-degree environment.
Walking in Others' Shoes
A final, more complex layer of situated reasoning is human-centric grounding. This occurs when an agent must understand a perspective that is not its own—for example, a robot following the command "pick up the cup on my left."
In Walk in Others' Shoes with a Single Glance, Yuqi Bu and David Hsu introduce the TEP method. It uses a top-view perspective transformation to bridge the gap between the robot's viewpoint and the human's. This allows the model to perform "mental rotation," essentially simulating what the world looks like from the speaker's position to resolve the spatial reference.
Why It Works
Situated reasoning works because it converts a high-dimensional, abstract problem (mapping language to a 3D coordinate sea) into a lower-dimensional, structured one (calculating geometry relative to a known origin). By incorporating orientation data—whether through explicit "situation" prompts in SQA3D or speaker orientation in LanguageRefer—models move from merely identifying objects to truly understanding the spatial context of the world they inhabit.