When humans interact with the world, we do not see the environment as a flat grid of pixels. If a camera angle shifts, or if a table is raised by a few inches, our understanding of an object's physical location remains completely unchanged. We operate within an intrinsic, metric mental map of three-dimensional space.
Traditional end-to-end robotic manipulation policies often bypass this mental map, attempting to map 2D camera images directly to motor commands. While simple to implement, these 2D pixel-space policies are notoriously fragile. A slight perturbation in camera placement, a change in table height, or even a flat photograph of an object can cause them to fail.
To overcome these limitations, a powerful paradigm has emerged: 3D spatial grounding. By lifting visual observations into explicit 3D representations—such as point clouds, voxel grids, or continuous 3D feature fields—and mapping them directly to metric 3D action spaces, researchers are building robots that inherently understand the geometry of their workspaces.
Viewpoint Invariance via Geometric Lifting
The primary flaw of a 2D vision-to-action policy is that it confounds the perspective of the sensor with the geometry of the task. If a camera is bumped or relocated, the target object's pixel coordinates change entirely, forcing a 2D network to relearn the task from the new perspective.
3D spatial grounding solves this by lifting visual features into a consistent 3D coordinate frame. As shown by Zhou Xian et al. in ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation, projecting visual features into a 3D point cloud representation allows robotic policies to achieve viewpoint invariance. Because the policy operates on the reconstructed 3D structure of the workspace rather than raw pixels, its performance remains stable even when camera positions change.
[2D Camera Images] ──(Lifting via Depth/Calibration)──> [3D Point Cloud / Feature Field] ──> [Consistent 3D Action]
▲
(Camera moves or bumps)
Wilbert Pumacay et al. systematically demonstrated this advantage in THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation, finding that models utilizing 3D spatial representations are significantly more robust to camera pose variations than their 2D counterparts. By decoupling task execution from the specific viewpoint of the sensor, the robot treats the workspace as a stable physical volume. This concept is pushed even further by Theophile Gervet et al. in Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation, which represents the workspace as a continuous 3D feature field. This inductive bias allows the robot to perform high-precision 6-DoF control that generalizes seamlessly across novel viewpoints.
Disambiguating Volume from Texture
Operating purely in 2D also leaves robots vulnerable to visual hallucinations. For example, a 2D vision model can easily confuse a flat photograph of a cup with a real, volumetric cup because their surface textures look identical from a single angle.
Explicit 3D grounding forces the model to reason about physical volume rather than surface appearance. In PointVLA: Injecting the 3D World into Vision-Language-Action Models, Chengmeng Li et al. demonstrated that integrating 3D geometric data into vision-language-action models prevents robots from attempting to interact with 2D representations of objects, such as photographs. The spatial awareness allows the system to distinguish between physical volume and flat visual textures.
Furthermore, grounding observations in absolute 3D space ensures that the model's predictions are tied to physical coordinates rather than relative pixel positions. This makes the policy highly resilient to vertical environmental shifts. As noted by Connor Schenck et al. in Learning the RoPEs: Better 2D and 3D Position Encodings with STRING, incorporating three-dimensional spatial geometry into position encodings enhances the robustness of robotic policies against out-of-distribution environmental shifts, such as changes in table height or elevations that typically cause 2D-based models to fail completely.
The Structural Shortcut: Why 3D is Sample Efficient
A common misconception is that processing 3D data is too computationally expensive or data-hungry compared to 2D images. In reality, 3D representations act as a massive structural shortcut for neural networks.
When a network is trained on 2D images, it must dedicate a significant portion of its parameter capacity to learning the implicit laws of physics, perspective, and depth projection from thousands of examples. By providing the model with explicit 3D geometric data (like point clouds), we eliminate the need for the model to "guess" depth.
2D Policy: [Pixels] ──(Network must learn perspective + depth + task)──> [Action] (Requires massive data)
3D Policy: [3D Coordinates] ──(Network only needs to learn the task)──> [Action] (Highly sample efficient)
As a result, 3D pre-training and imitation learning demonstrate extreme sample efficiency. Haoyi Zhu et al. showed in Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning that prioritizing spatial coordinate information over visual color data allows models to achieve high performance while requiring orders of magnitude less data than 2D image pre-training.
This sample efficiency becomes incredibly potent when paired with generative frameworks like diffusion models. Yanjie Ze et al. demonstrated in 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations that using 3D point clouds as the primary input for imitation learning enables robots to generalize to entirely new spatial configurations with significantly fewer human demonstrations than traditional 2D methods.
Unifying Perception and Action in Metric Space
The final piece of the 3D spatial grounding puzzle is closing the loop: mapping 3D perception directly to a metric 3D action space.
Instead of predicting actions as 2D pixel vectors (e.g., "move toward pixel coordinate x,y"), spatially grounded policies predict metric 3D trajectories or 6-DoF poses. Hanzhi Chen et al. demonstrated in VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation that representing robotic actions as metric 3D trajectories rather than 2D pixel-space vectors significantly improves success rates in complex manipulation tasks. This spatial awareness allows robots to accurately execute non-linear movements, such as opening drawers or lifting lids, which require precise depth perception.
This metric mapping is most elegant when the robot's own body or gripper is integrated directly into the 3D representation. Yufei Wang showed in One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments that mapping 3D points directly to 3D actions through a dense transformation architecture preserves critical spatial relationships by allowing the robot to execute actions based on specific points in the point cloud, such as the location of its own gripper.
Unified Point Cloud:
[ Object Points (• • • •) ] <─── Direct Spatial Reasoning ───> [ Gripper Points (▲ ▲) ]
By treating both the environment and the robot's end-effector as part of the same unified geometric space, the network can directly reason about spatial interactions. As pioneered by Arsalan Mousavian et al. in 6-DOF GraspNet: Variational Grasp Generation for Object Manipulation and extended by Adithyavairavan Murali et al. in Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping, processing 3D point clouds that include both the object geometry and the gripper's control points allows the network to learn the specific spatial interactions required for a stable and functional grasp. The tool and the target are evaluated together, making collision avoidance and precise geometric alignment a natural byproduct of the architecture.
Key Foundational Work
Learning 6-DOF Grasping Interaction via Deep Geometry-Aware 3D Representations (2018) — contributed 1 concept elements to this direction.
Data-Driven Grasp Synthesis - A Survey (2013) — contributed 1 concept elements to this direction.