OTOcc: Optimal Transport for Occupancy Prediction

Pengteng Li, Ying He, F. Richard Yu, Pinhao Song, Xingchen Zhou, Guang Zhou, Xiaobai Li

Abstract

Accurately projecting 2D camera data into a 3D world is a fundamental challenge in autonomous driving. Existing depth-based and query-based methods often struggle with depth estimation errors or high computational costs due to data sparsity. This paper introduces OTOcc, a framework presented at IJCAI-24 that re-imagines 2D-to-3D semantic mapping as an Optimal Transport (OT) problem. By treating pixels as "suppliers" and 3D voxels as "demanders," OTOcc utilizes the Sinkhorn-Knopp algorithm to find a mathematically optimal mapping. Experimental results on the Occ3D-nuScenes benchmark show that OTOcc achieves state-of-the-art accuracy (29.10% mIoU) while reducing computational overhead by 4.58%.

The Challenge of 2D-to-3D Mapping

For autonomous vehicles, understanding the 3D occupancy of the surrounding environment is critical for safe navigation. However, mapping flat 2D images to a 3D grid is notoriously difficult because the 3D world is inherently sparse—most of it is empty space. Current approaches typically fall into two categories:

  1. Depth-based methods: These estimate pixel depth to place information in 3D. However, if the depth estimate is slightly off, the resulting 3D model is significantly degraded.
  2. Query-based methods: These use heavy Transformer architectures with attention mechanisms to let 3D voxels "look" for relevant pixels. While powerful, this is computationally expensive and often struggles with the inherent sparsity of the data.
Comparison of 2D-to-3D mapping paradigms: (a) Depth-based, (b) Query-based, and (c) the proposed OT-based method.

Methodology: Optimal Transport for Semantic Conversion

OTOcc proposes a resource allocation perspective to solve the mapping problem. Instead of guessing depth or using brute-force attention, it treats the conversion as a mathematical optimization task.

Pixels as Suppliers, Voxels as Demanders

In this framework, each 2D pixel is viewed as a supplier of "semantic marks," while each 3D voxel is a demander that needs these marks to determine its state. The "cost" of transporting information from a pixel to a voxel is defined by a weighted occupancy prediction loss, aiming to find a mapping that minimizes the error in the final 3D prediction.

Efficient Optimization with Sinkhorn-Knopp

To solve the transport problem efficiently during training, OTOcc employs the Sinkhorn-Knopp algorithm. This iterative process finds an optimal mapping matrix ($\pi^*$) without the massive overhead of standard linear programming. A key advantage of this approach is that the complex OT optimization happens during training. Once the model is trained, the "Strategy Optimization" module is discarded, and the learned mapping is used directly. This makes OTOcc faster during actual inference than many Transformer-based rivals.

The OT-based method uses Sinkhorn-Knopp iterations to minimize transportation cost and find the optimal mapping $\pi^*$.

Architectural Enhancements: Block Reading and MPV

To maintain a low memory footprint and handle high-resolution data, OTOcc utilizes Block Reading, dividing the 3D space into manageable blocks. It also leverages Multi-Perspective View (MPV) features (front, side, top, etc.) to ensure the model captures fine-grained details without overwhelming the GPU. An Adaptive Fusion Decoder then merges these multi-perspective features into the final 3D occupancy grid.

Experimental Results

OTOcc was evaluated on industry-standard benchmarks, including Occ3D-nuScenes and nuScenes.

State-of-the-Art Performance

The model achieved a new state-of-the-art 29.10% mIoU (mean Intersection over Union) on the Occ3D-nuScenes benchmark, outperforming previous leaders such as CTF-Occ and TPVFormer.

Efficiency and Precision

Beyond accuracy, OTOcc demonstrates a 4.58% reduction in computational overhead compared to other state-of-the-art methods. Qualitatively, the model is significantly better at detecting small or "confusing" objects. For instance, it achieved 25.72% accuracy for motorcycles and 23.81% accuracy for pedestrians, categories that are often missed or blurred by traditional depth-based methods.

Conclusion

By replacing heuristic or brute-force mapping with a mathematically grounded Optimal Transport approach, OTOcc provides a more stable and efficient solution for 3D occupancy prediction. Its ability to handle sparsity through calculated mathematical optimums rather than noisy learned attention marks a significant step forward for autonomous perception. Future work will focus on developing even more efficient OT solvers and exploring more powerful decoders to further refine 3D voxel representations.

Summary updated

BibTeX
@inproceedings{li2024otocc,
  author = {Pengteng Li and Ying He and F. Richard Yu and Pinhao Song and Xingchen Zhou and Guang Zhou and Xiaobai Li},
  title = {{OTOcc: Optimal Transport for Occupancy Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=ofmlpH5dDl},
}
Figure