Abstract
Accurately projecting 2D camera data into a 3D world is a fundamental challenge in autonomous driving. Existing depth-based and query-based methods often struggle with depth estimation errors or high computational costs due to data sparsity. This paper introduces OTOcc, a framework presented at IJCAI-24 that re-imagines 2D-to-3D semantic mapping as an Optimal Transport (OT) problem. By treating pixels as "suppliers" and 3D voxels as "demanders," OTOcc utilizes the Sinkhorn-Knopp algorithm to find a mathematically optimal mapping. Experimental results on the Occ3D-nuScenes benchmark show that OTOcc achieves state-of-the-art accuracy (29.10% mIoU) while reducing computational overhead by 4.58%.
The Challenge of 2D-to-3D Mapping
For autonomous vehicles, understanding the 3D occupancy of the surrounding environment is critical for safe navigation. However, mapping flat 2D images to a 3D grid is notoriously difficult because the 3D world is inherently sparse—most of it is empty space. Current approaches typically fall into two categories:
- Depth-based methods: These estimate pixel depth to place information in 3D. However, if the depth estimate is slightly off, the resulting 3D model is significantly degraded.
- Query-based methods: These use heavy Transformer architectures with attention mechanisms to let 3D voxels "look" for relevant pixels. While powerful, this is computationally expensive and often struggles with the inherent sparsity of the data.
Methodology: Optimal Transport for Semantic Conversion
OTOcc proposes a resource allocation perspective to solve the mapping problem. Instead of guessing depth or using brute-force attention, it treats the conversion as a mathematical optimization task.
Pixels as Suppliers, Voxels as Demanders
In this framework, each 2D pixel is viewed as a supplier of "semantic marks," while each 3D voxel is a demander that needs these marks to determine its state. The "cost" of transporting information from a pixel to a voxel is defined by a weighted occupancy prediction loss, aiming to find a mapping that minimizes the error in the final 3D prediction.
Efficient Optimization with Sinkhorn-Knopp
To solve the transport problem efficiently during training, OTOcc employs the Sinkhorn-Knopp algorithm. This iterative process finds an optimal mapping matrix ($\pi^*$) without the massive overhead of standard linear programming. A key advantage of this approach is that the complex OT optimization happens during training. Once the model is trained, the "Strategy Optimization" module is discarded, and the learned mapping is used directly. This makes OTOcc faster during actual inference than many Transformer-based rivals.
Architectural Enhancements: Block Reading and MPV
To maintain a low memory footprint and handle high-resolution data, OTOcc utilizes Block Reading, dividing the 3D space into manageable blocks. It also leverages Multi-Perspective View (MPV) features (front, side, top, etc.) to ensure the model captures fine-grained details without overwhelming the GPU. An Adaptive Fusion Decoder then merges these multi-perspective features into the final 3D occupancy grid.
Experimental Results
OTOcc was evaluated on industry-standard benchmarks, including Occ3D-nuScenes and nuScenes.
State-of-the-Art Performance
The model achieved a new state-of-the-art 29.10% mIoU (mean Intersection over Union) on the Occ3D-nuScenes benchmark, outperforming previous leaders such as CTF-Occ and TPVFormer.
Efficiency and Precision
Beyond accuracy, OTOcc demonstrates a 4.58% reduction in computational overhead compared to other state-of-the-art methods. Qualitatively, the model is significantly better at detecting small or "confusing" objects. For instance, it achieved 25.72% accuracy for motorcycles and 23.81% accuracy for pedestrians, categories that are often missed or blurred by traditional depth-based methods.
Conclusion
By replacing heuristic or brute-force mapping with a mathematically grounded Optimal Transport approach, OTOcc provides a more stable and efficient solution for 3D occupancy prediction. Its ability to handle sparsity through calculated mathematical optimums rather than noisy learned attention marks a significant step forward for autonomous perception. Future work will focus on developing even more efficient OT solvers and exploring more powerful decoders to further refine 3D voxel representations.
Summary updated