OTOcc: Optimal Transport for Occupancy Prediction

Pengteng Li, Ying He, F. Richard Yu, Pinhao Song, Xingchen Zhou, Guang Zhou

Abstract

The autonomous driving community is highly interested in 3D occupancy prediction due to its outstanding geometric perception and object recognition capabilities. However, previous methods are limited to existing semantic conversion mechanisms for solving sparse ground truths problem, causing excessive computational demands and sub-optimal voxels representation. To tackle the above limitations, we propose OTOcc, a novel 3D occupancy prediction framework that models semantic conversion from 2D pixels to 3D voxels as Optimal Transport (OT) problem, offering accurate semantic mapping to adapt to sparse scenarios without attention or depth estimation. Specifically, the unit transportation cost between each demander (voxel) and supplier (pixel) pair is defined as the weighted occupancy prediction loss. Then, we utilize the Sinkhorn-Knopp Iteration to find the best mapping matrices with minimal transportation costs. To reduce the computational cost, we propose a block reading technique with multi-perspective feature representation, which also brings fine-grained scene understanding. Extensive experiments show that OTOcc not only has the competitive prediction performance but also has about more than 4.58% reduction in computational overhead compared to state-of-the-art methods.

Introduction: The Challenge of 3D Occupancy

In autonomous driving, 3D occupancy prediction aims to classify every "voxel" (a 3D pixel) around a vehicle as occupied or free. This is crucial for detecting irregularly shaped objects that standard 3D bounding boxes might miss. However, mapping 2D camera data into a 3D grid is difficult. Current methods usually fall into two camps:

  1. Depth-based: These methods estimate depth for every pixel to "project" features into 3D. If the depth estimate is inaccurate, the resulting 3D model is compromised.
  2. Query-based: These methods use attention mechanisms (like Transformers) to let 3D voxels search for relevant 2D pixels. This is computationally expensive and often struggles with the inherent sparsity of 3D space.

OTOcc identifies a need for a more efficient, mathematically grounded way to map semantics from 2D to 3D without relying on these heavy or fragile components.

Methodology: Mapping via Optimal Transport

The core innovation of OTOcc is treating 2D-to-3D mapping as an Optimal Transport (OT) problem. Instead of guessing the mapping through a neural network, the model derives it mathematically.

Comparison of Depth-based, Query-based, and the proposed OT-based mapping methods.
  • Pixels as Suppliers, Voxels as Demanders: The model treats each 2D pixel as a "supplier" of semantic information and each 3D voxel as a "demander" that needs that information.
  • Sinkhorn-Knopp Iteration: To find the most efficient way to "transport" information from pixels to voxels, the model uses the Sinkhorn-Knopp algorithm. It calculates a mapping matrix that minimizes a "transportation cost," defined as the weighted occupancy prediction loss.
  • Efficiency through Inference Pruning: While the Sinkhorn-Knopp optimization is used during training to find the best mapping, it is discarded during inference. The model "remembers" the optimized mapping via a simple MLP, making it faster when deployed on a vehicle.
  • Multi-Perspective View (MPV): Moving from standard Tri-Perspective Views (3 planes) to MPV (5 planes—front, left, right, rear, and top) allows the model to capture finer details of the scene, especially for complex objects.

Key Results

OTOcc was evaluated on the Occ3D-nuScenes benchmark and the nuScenes LiDAR Segmentation task. Key findings include:

  • SOTA Performance: OTOcc achieved 29.10% mIoU on the Occ3D-nuScenes benchmark, outperforming previous leaders like CTF-Occ and TPVFormer.
  • Computational Gains: The method reduced computational overhead by more than 4.58% compared to state-of-the-art methods. During inference, OTOcc used only 5.3G of memory, significantly lower than TPVFormer-Base (7.1G).
  • Small Object Detection: Qualitatively, the model showed a superior ability to detect small or blurry objects, such as traffic cones and pedestrians, which are often missed by depth-based or attention-based models.

Conclusion

OTOcc introduces a novel framework for 3D occupancy prediction that replaces heavy attention mechanisms and error-prone depth estimation with Optimal Transport theory. By modeling the conversion of 2D image pixels to 3D voxels as a transportation problem, the authors achieve state-of-the-art accuracy while significantly reducing computational overhead. Future work will focus on further optimizing the training stability and exploring more powerful decoders to leverage the OT-mapped features.

Summary updated

Abstract

The autonomous driving community is highly interested in 3D occupancy prediction due to its outstanding geometric perception and object recognition capabilities. However, previous methods are limited to existing semantic conversion mechanisms for solving sparse ground truths problem, causing excessive computational demands and sub-optimal voxels representation. To tackle the above limitations, we propose OTOcc, a novel 3D occupancy prediction framework that models semantic conversion from 2D pixels to 3D voxels as Optimal Transport (OT) problem, offering accurate semantic mapping to adapt to sparse scenarios without attention or depth estimation. Specifically, the unit transportation cost between each demander (voxel) and supplier (pixel) pair is defined as the weighted occupancy prediction loss. Then, we utilize the Sinkhorn-Knopp Iteration to find the best mapping matrices with minimal transportation costs. To reduce the computational cost, we propose a block reading technique with multi-perspective feature representation, which also brings fine-grained scene understanding. Extensive experiments show that OTOcc not only has the competitive prediction performance but also has about more than 4.58% reduction in computational overhead compared to state-of-the-art methods.
BibTeX
@inproceedings{li2024otocc,
  author = {Pengteng Li and Ying He and F. Richard Yu and Pinhao Song and Xingchen Zhou and Guang Zhou},
  title = {{OTOcc: Optimal Transport for Occupancy Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=ofmlpH5dDl},
}
Figure