Traditional computer vision and robotics have long relied on simplified representations of the 3D world. Whether through 2.5D depth maps, which capture only the visible surfaces from a single perspective, or 2D Bird’s-Eye View (BEV) projections, which flatten the world into a top-down map, these methods inevitably discard crucial spatial information.
A growing body of research argues that to truly understand and navigate complex environments, agents must move toward 3D volumetric occupancy representations. By quantizing space into a structured grid of voxels, these models preserve the full geometric and semantic context of a scene, including occlusions, vertical structures, and irregular shapes.
The Problem of Dimensionality Collapse
The primary motivation for volumetric modeling is the failure of "flattened" representations in complex environments. In autonomous driving, BEV maps are often used to simplify planning. However, as Huang et al. argue, flat maps struggle with off-ground objects, overpasses, and uneven terrain. Similarly, in indoor robotics, Yu et al. note that preserving the vertical axis is essential for distinguishing between objects at different heights, such as a shelf versus a floor-standing cabinet.
This "information collapse" is particularly problematic for legged robots. Frey et al. and Miki et al. demonstrate that traditional 2.5D elevation maps—which store only the highest point at each $(x, y)$ coordinate—render robots "blind" to overhanging obstacles like tunnels, tables, or low ceilings. By adopting a 3D voxel space, robots can recognize traversable paths beneath obstacles that would otherwise be classified as impassable walls.
Beyond Bounding Boxes
In addition to preserving verticality, volumetric representations offer a more precise alternative to the 3D bounding boxes commonly used in object detection. While a box provides a coarse estimate of an object's location, it often includes significant empty space and fails to capture irregular geometries.
Sima shows that replacing bounding boxes with dense 3D occupancy grids can reduce collision rates in motion planning by up to 58%, as the grid captures the exact boundaries of obstacles. This precision is further extended by Chen et al., who introduce "3D occupancy grounding." This task requires a model to identify the specific voxel-level shape of an object based on natural language, allowing for the localization of complex structures like construction vehicles where traditional boxes fail.
Furthermore, Zheng et al. demonstrate that 3D semantic occupancy—where each voxel is assigned a category label—serves as a powerful "world model." This allows autonomous systems to predict not just where an object is, but how the entire 3D geometry of a scene will evolve over time.
A Unified Landscape for Reasoning
Voxel grids do more than just store geometry; they provide a structured "canvas" for multimodal reasoning. Because voxels are discrete and spatially consistent, they can integrate data from various sources:
- Sensor Fusion: Xu et al. highlight how volumetric grids allow for the seamless fusion of LiDAR, cameras, and radar, providing a continuous semantic understanding even in regions not directly hit by sensors.
- Language and Logic: Xu et al. and Fan et al. have integrated Large Language Models (LLMs) with occupancy grids. By "seeing" in 3D, these models can generate more accurate navigation instructions and reduce landmark hallucinations by maintaining a holistic view of the environment's structure.
- Robotic Manipulation: Huang et al. use 3D voxel grids to represent "value maps," where different regions of space are assigned rewards or costs based on language instructions. This allows a robot to treat a complex task—like "avoid the laptop while reaching for the mug"—as a simple pathfinding problem through a 3D cost landscape.
Efficiency and Scalability
While 3D grids are computationally more expensive than 2D maps, recent work has focused on making them scalable. Wang et al. introduced the OpenOccupancy benchmark to standardize large-scale 360-degree semantic occupancy perception. To handle the increased data density, Zhou 0001 et al. proposed "vertical slice" approaches that model height-axis hierarchies more efficiently, while Malagón et al. developed high-performance voxel engines to speed up reinforcement learning in these environments.
By shifting from surface-level observations to dense volumetric understanding, the field is moving toward agents that don't just see images, but truly comprehend the three-dimensional structure of the world they inhabit.
Key Foundational Work
Structure from Motion with Objects (2016) — contributed 1 concept elements to this direction.
A Linear Approach to Matching Cuboids in RGBD Images (2013) — contributed 1 concept elements to this direction.
RGBD Datasets: Past, Present and Future (2016) — contributed 1 concept elements to this direction.