Robots and autonomous systems perceive the world in three dimensions, capturing dense point clouds, sonar echoes, and stereo depth maps. However, processing full 3D volumetric data—such as voxel grids—is computationally punishing. Volumetric representations scale cubically ($O(N^3)$) in memory and time, creating a severe bottleneck for real-time edge devices like mobile robots, drones, and augmented reality (AR) headsets.
To bypass this "calculation explosion," researchers exploit a fundamental physical truth about most environments: the world is structurally asymmetric. While terrain, obstacles, and objects vary complexly across the horizontal plane, their vertical structure is often highly regular or bounded by gravity.
By projecting 3D spatial data into 2D bird's-eye-view (BEV) grids or compressing it into 2.5D heightmaps, we can drop a geometric dimension. This dimensionality reduction unlocks highly optimized 2D algorithms and image-processing pipelines while retaining the critical spatial fidelity required to navigate, plan, and interact with the 3D world.
Compressing Volumetric Space into 2.5D Heightfields
A 2.5D heightfield or elevation grid simplifies 3D space by storing a continuous height value ($Z$) over a discrete 2D coordinate grid ($X, Y$). Instead of tracking whether every individual cubic centimeter of a room is full or empty, the system only tracks the upper boundary of the terrain or object surface.
Full 3D Voxel Grid (Cubic Cost) 2.5D Heightfield (Quadratic Cost)
[ ] [ ] [ ] [ ]
[ ] [X] [X] [ ] | | 2 | 2 | | <- Height values
[X] [X] [X] [X] | 1 | 2 | 2 | 1 |
This structural prior drastically reduces computational overhead. For instance, in mobile augmented reality, Watson et al. developed a real-time scene reconstruction framework that replaces expensive 3D convolutions with top-down 2.5D heightfields. This allows mobile devices to reconstruct indoor geometry at 30 frames per second because the architecture assumes vertical geometry remains relatively simple compared to horizontal layout changes.
The same principle applies to physical interaction with complex materials. Modeling individual particles of granular media (like sand, soil, or beans) is impossible in real time. To solve this, Schenck and Kreis et al. represented granular states as compact 2.5D heightmaps. Capturing the top-down volumetric distribution of the media provides a sufficient state space for reinforcement learning models to successfully learn scooping, dumping, and shaping behaviors.
For dynamic locomotion, the authors integrated local 2.5D height scans directly into the end-to-end control policy of a quadrupedal robot. By treating individual grid cells as localized height values relative to the robot's base frame, the policy evaluates collision threats and foot placement without the overhead of a full 3D volumetric map.
Top-Down Projections for Accelerated Perception
Beyond saving memory, flattening 3D data into 2D representations allows researchers to leverage mature, hardware-accelerated computer vision architectures. 2D Convolutional Neural Networks (CNNs) and image-matching algorithms run significantly faster than their 3D counterparts.
A common strategy is to perform semantic or geometric reasoning in 2D before re-projecting the results back into 3D space:
- 2D Image-Space Segmentation: Kim et al. and Wang et al. project raw 3D point clouds into 2D image planes to perform semantic segmentation via lightweight 2D networks. Once labeled in 2D, the pixels are projected back into 3D coordinates using camera intrinsics, bypassing direct 3D point cloud analysis entirely.
- Rasterized Mapping: Ali et al. convert 3D LiDAR data into a collection of 2D rasterized images. By treating local maps as images, they utilize high-speed image-matching techniques (such as Bag of Visual Words) to achieve accurate loop closure in lifelong SLAM with minimal CPU usage.
- Spatial Utility Maps: For autonomous exploration, Wang et al. project 3D environmental data into 2D utility maps. This abstraction allows a lightweight 2D CNN to efficiently decide where a robot should point its camera next to map an unknown room.
This projection strategy is highly versatile. Zecchin et al. transform 3D point clouds into 2D top-view grids to efficiently calculate mmWave beam selection for vehicle-to-infrastructure networks. Similarly, Li et al. project 3D point clouds onto a 2D reference plane (such as a conveyor belt) to simplify complex spatial luggage measurements into fast, real-time edge-computing tasks.
Bridging 2D Planning to 3D Action
If an autonomous agent compresses its map into a 2D or 2.5D representation, how does it execute physically sound actions in a 3D world? The answer lies in mathematical mapping functions that bridge the lower-dimensional planning space to higher-dimensional physical states.
In the earliest days of robotics, Lewis et al. demonstrated that projecting laser rangefinder data onto a discrete 2D grid was sufficient to classify terrain into traversable, hazardous, or occluded regions for obstacle avoidance. Modern systems refine this by embedding specific physical constraints into the projection logic.
For instance, Wu et al. project 3D spatial data onto a 2D navigation plane based on the robot's specific height and width. This allows standard 2D path planners to successfully negotiate height-restricted obstacles, like low-clearance arches, without requiring a full 3D navigation stack.
3D Obstacle (Low Arch) ----> Filtered by Robot Profile ----> 2D Navigation Grid
_______ [X] [X] [X] (Blocked)
/ \ <- Too low for robot
/ \ <- Safe clearance [ ] [ ] [ ] (Free)
When navigating rugged, multi-level terrains like stairs or sills, Gutmann and Zhang et al. utilize 2.5D elevation grids to calculate "trafficability factors" based on slope and roughness. This allows humanoid robots or vehicles to find realistic paths across uneven surfaces without traversing full 3D voxel maps.
To execute smooth, high-speed trajectories on uneven terrain, Xu et al. use a "terrain pose mapping" function. The robot plans its trajectory in a computationally efficient 2D space, and the system dynamically projects each 2D state into a full 3D pose by performing iterative plane-fitting on the local point cloud corresponding to the robot's footprint.
By decoupling the dimensionality of representation from the dimensionality of action, these frameworks achieve the best of both worlds: the speed of 2D computation and the geometric accuracy of 3D execution.
Key Foundational Work
Winning the AAAI Robot Competition (1993) — contributed 1 concept elements to this direction.
A Scanning Laser Rangefinder for a Robotic Vechicle (1977) — contributed 1 concept elements to this direction.